Stress Tests with Workload Balancing

TL;DR. This post highlights that workload distribution for stress tests does not need traditional workload distribution approaches, but rather given the simpler requirements, we can implement a simpler scheme.

Introduction

To stress test our input APIs and data transformations, a fleet of simulators pounds our processing nodes looking for bottlenecks and bugs. Each simulator is deployed as Kubernetes job. We can easily start and stop these jobs, and we can also easily tune their parameters. For large scale testing, we have multiple replicas of a particular job run simultaneously.

Clearly, standard workload distribution approaches can be applied. E.g. we could treat all the replicas as a hash ring. A leader would be elected to distribute the load to workers. In the event of a failure, the leader would be responsible for redistributing the load appropriately.

Leader workflow

Replicas wait for work. After completing their assignments, they would ask the leader for the next job. If the leader is no longer available, another election is performed.

Replica workflow

This procedure requires that each replica do several things alongside running the simulation.

1. Act as a leader or replica
2. Communicate with other replicas in case something goes wrong
3. Determine whether a replica/leader has timed out or is simply busy

Simple Constraints for Stress Tests

Our problem has more relaxed constraints which allowed us to come up with a less complex solution.

1. All test data is based on the ID of the replica running a simulation. This eliminates the need for a leader to distribute work and monitor the progress of individual replicas. If a replica is aware of how many others are in the ring and of the total work to be done, the correct data can be generated.
2. Our simulations are only tests. We aren’t concerned with correctly processing incomplete jobs if a replica failed. As long as we know how much data was sent from a replica before it died, we know how much should have been processed by our network.
3. Our extended simulations are run over a long period of time. There is no reason to distribute work up front. Since data is generated over and over, a check can be performed to see how many replicas are available, and which work should be done by a particular replica.

Simple(r) Solution

Our constraints allow us to reduce the complexity of each replica. We can conflate the leader/worker nature of our ring because all of our data is deterministic. This eliminates a worker’s need for a leader to assign work. If a replica knows its ID, it can correctly generate its own workload.

When a stress test replica is ready, it creates an ephemeral znode in ZooKeeper. These znodes are also marked sequential.

// Join will create an ephermeral node under the root znode
func (g *Group) Join() error {
  var err error
  path := fmt.Sprintf("%s/guid_", g.root)
  node, err := g.conn.Create(
    path, 
    nil, 
    zk.FlagEphemeral|zk.FlagSequence, 
    zk.WorldACL(zk.PermAll)
  )
  
  if err != nil {
    if err != zk.ErrNoNode {
      return err
    }
  }
  g.node = strings.TrimPrefix(node, g.root+"/")
  return nil
}

Once a replica has joined the group, a list of all ephemeral znodes is obtained and sorted.

// Position returns the position of the current replica in the group
func (g *Group) Position() int {
  children, _, err := g.conn.Children(g.root)
  if err != nil {
    glog.Error(err)
    return -1
  }
  sort.Strings(children)
  for i, child := range children {
    if child == g.node {
      return i
    }
  }
  return -1
}

Based on the number of znodes, and the replica’s position in the list, the correct work can be generated as many times as necessary.

for {
  node := group.Position()
  totalNodes := group.Members()
  rangeSize := totalPhones / totalNodes
  startRange := node * rangeSize
  sendIds(startRange, startRange + rangeSize)
}

With each iteration, the total members and position are updated. The test data is regenerated.