Load Testing a Service with ~20,000 Requests per Second with Locust, Helm, and Kustomize
At TeamSnap, it’s important to us to be sure our services can handle all of our users’ traffic reliably. We can do that in a stress-free way by creating artificial traffic in a staging environment with Locust.io, and using Kubernetes to scale up our Locust workers as much as we need to generate the traffic.
In this article, I’ll go over the tools our platform team uses to make load testing easy enough to configure and spin up at scale that our developers are enabled without needing a background in infrastructure.
Tools Overview
Terraform
Terraform is a HashiCorp tool for automating infrastructure creation with declarative code.
Helm
Helm describes itself as “the package manager for Kubernetes”, and it’s great for deploying common projects like Locust without needing to look at any infrastructure code.
Kustomize
Kustomize is a tool built into kubectl that’s capable of making arbitrary changes to Kubernetes manifests, and comes with some useful helpers like declarative config map generators (which we’ll make use of here).
Locust
Locust.io is a modern load testing tool that lets you write a simple python script (example here) to define load testing behavior and which end points to hit. Locust is a great choice because it can distribute load testing across horizontally scalable worker nodes, which combined with Kubernetes, means we can easily scale up resources to generate a huge amount of requests/second against whatever service we want to test.
Strategy for combining everything
Terraform is our go-to tool for building infrastructure, so it was an easy choice to use that to build our autoscaling Kubernetes cluster.
The easiest method we’ve found to deploy Locust is with deliveryhero’s locust Helm chart. Even using the Helm chart to generate all the Kubernetes manifests we need to deploy locust, there are some manual steps in the README for creating config maps with your custom load test scripts. We can stack Kustomize on top of Helm to automate those steps and end up with a single declarative config file that manages everything.
Solution
First, we need to create our Kubernetes cluster. This example uses the terraform-google-modules/kubernetes-engine/google module to create the cluster, but using the Google provider’s container_cluster resource directly will work great too. The import things to configure are the node pool’s machine type and an autoscaling min and max node count.
This following code declares an n1-standard-4 machine type node pool (4 CPUs and 15GB memory per node), and a max node count of 50.
Next, lets take a look at how we’re going to deploy Locust to the cluster.
Looking through the config options in the Locust chart’s README, these are the values related to the Locust workers that we’ll need to configure for our cluster. If we configure the worker pod resources to a size where we’re able to fit 2 workers on each Kubernetes node, then when we enable the horizontal pod autoscaler, 100 replicas should fit in our 50 node k8s cluster.
As for config options related to our custom locustfile and load testing, this next snippet contains the values we’ll set. The important thing to note here is that this Helm chart expects us to provide config maps that contain our custom locustfile as well as (optionally) any additional files that the locustfile requires.
At this point, we could just run helm install with those value overrides, but there’s a helmCharts field in kustomize we can leverage that’ll let us declare the Helm chart repo, name, version, and values all in the kustomization.yaml which lets us easily version the config in git alongside our application code. When we run kubectl kustomize it will download the Helm chart and run helm template with the values we specified.
Now we have the equivalent of what’s in the Helm chart README and we’d still need to manually create the necessary config maps by running imperative kubectl create cm... commands. However, now that we’re stacking kustomize on top of Helm, we can use a configMapGenerator instead.
Assuming a directory structure that looks like this:
.
├── kustomization.yaml
├── lib
│ ├── payload.json
│ ├── payload2.json
│ └── payload3.json
└── my-locustfile.pyWe can create the config maps declaratively by adding the following snippet to our kustomization.yaml.
Putting it all together in this last gist, when we run kubectl kustomize --enable-helm, it’ll generate all the k8s manifests contained in the Helm chart as well as creating the additional config maps we need to run our load tests. To deploy the manifests, we can pipe the output into kubectl apply. The full deploy command is kubectl kustomize --enable-helm | kubectl apply -f - .
Since kustomize adds a hash suffix to the end of config map names by default, this has an additional benefit in that if we make changes to our locustfile and re-run our deploy command, the locust worker deployment will see that there was an update and automatically spin up new worker pods with the updated config.
To view the Locust UI, we can use port forwarding (kubectl port-forward service/load-test-locust 8089:8089) to make it available in a browser at localhost:8089. Locust has a great guide for getting started , so I’m not going to go into usage details, but with the Kubernetes config and the kustomization.yaml described in this article, we were able to easily generate 19,000+ requests per second with 80 worker pods for our load test and we were far from reaching the limits.
