Stress Tests with Workload Balancing
TL;DR. This post highlights that workload distribution for stress tests does not need traditional workload distribution approaches, but rather given the simpler requirements, we can implement a simpler scheme.
Introduction
To stress test our input APIs and data transformations, a fleet of simulators pounds our processing nodes looking for bottlenecks and bugs. Each simulator is deployed as Kubernetes job. We can easily start and stop these jobs, and we can also easily tune their parameters. For large scale testing, we have multiple replicas of a particular job run simultaneously.
Clearly, standard workload distribution approaches can be applied. E.g. we could treat all the replicas as a hash ring. A leader would be elected to distribute the load to workers. In the event of a failure, the leader would be responsible for redistributing the load appropriately.
Replicas wait for work. After completing their assignments, they would ask the leader for the next job. If the leader is no longer available, another election is performed.
This procedure requires that each replica do several things alongside running the simulation.
- Act as a leader or replica
- Communicate with other replicas in case something goes wrong
- Determine whether a replica/leader has timed out or is simply busy
Simple Constraints for Stress Tests
Our problem has more relaxed constraints which allowed us to come up with a less complex solution.
- All test data is based on the ID of the replica running a simulation. This eliminates the need for a leader to distribute work and monitor the progress of individual replicas. If a replica is aware of how many others are in the ring and of the total work to be done, the correct data can be generated.
- Our simulations are only tests. We aren’t concerned with correctly processing incomplete jobs if a replica failed. As long as we know how much data was sent from a replica before it died, we know how much should have been processed by our network.
- Our extended simulations are run over a long period of time. There is no reason to distribute work up front. Since data is generated over and over, a check can be performed to see how many replicas are available, and which work should be done by a particular replica.
Simple(r) Solution
Our constraints allow us to reduce the complexity of each replica. We can conflate the leader/worker nature of our ring because all of our data is deterministic. This eliminates a worker’s need for a leader to assign work. If a replica knows its ID, it can correctly generate its own workload.
When a stress test replica is ready, it creates an ephemeral znode in ZooKeeper. These znodes are also marked sequential.
// Join will create an ephermeral node under the root znode
func (g *Group) Join() error {
var err error
path := fmt.Sprintf("%s/guid_", g.root)
node, err := g.conn.Create(
path,
nil,
zk.FlagEphemeral|zk.FlagSequence,
zk.WorldACL(zk.PermAll)
)
if err != nil {
if err != zk.ErrNoNode {
return err
}
} g.node = strings.TrimPrefix(node, g.root+"/") return nil
}
Once a replica has joined the group, a list of all ephemeral znodes is obtained and sorted.
// Position returns the position of the current replica in the group
func (g *Group) Position() int {
children, _, err := g.conn.Children(g.root)
if err != nil {
glog.Error(err)
return -1
} sort.Strings(children)
for i, child := range children {
if child == g.node {
return i
}
} return -1
}
Based on the number of znodes, and the replica’s position in the list, the correct work can be generated as many times as necessary.
for {
node := group.Position()
totalNodes := group.Members()
rangeSize := totalPhones / totalNodes
startRange := node * rangeSize sendIds(startRange, startRange + rangeSize)
}
With each iteration, the total members and position are updated. The test data is regenerated.