Distributed Locust Testing on Google Kubernetes Engine (GKE)

A complete guide for running distributed Locust tests on Google Kubernetes Engine (GKE).

Google Cloud Platform + Locust

Introduction

Locust.io is an open-source tool which enables you to run performance tests against applications. One of the advantages of Locust is that it gives you the ability to distribute your performance tests across multiple machines so you can generate an even greater load on your application.

In this tutorial we’ll take a look at deploying Locust to Google Kubernetes Engine (GKE) in order to distribute load generation using a Master and Worker configuration.

Requirements

• All you need to follow this guide is access to a Google Cloud Platform (GCP) account. We will be using Google Kubernetes Engine and a few other GCP services as part of this guide.

Creating your Kubernetes Cluster

First we’ll need to get setup with a Kubernetes cluster. You need to sign into your Google Cloud Platform account and navigate to the Kubernetes Engine section of the console as seen below.

After you have reached this page you then need to:

• Select the ‘Create’ button.
• Click the ‘Configure’ button beside the ‘GKE Standard’ option.
• Choose a name for your cluster such as ‘distributed-locust-testing’ for example.
• Select the most appropriate zone for your cluster e.g. ‘europe-west1-c’.
• Hit the ‘Create’ button at the bottom of the screen.

You do not need to configure any other settings such as Automation, Networking, Security, etc. After a few minutes your Kubernetes cluster should have finished building and will be visible in your console.

You then need to:

• Click the name of your cluster under the ‘Name’ column.
• Select the ‘Connect’ button on this page.
• Then click on the ‘Run in Cloud Shell’ button.

You will then have your Cloud Shell opened with a command similar to this in your window:

$ gcloud container clusters get-credentials distributed-locust-testing --zone europe-west1-c --project infra-agent-999999

Once you hit ‘Enter’ you will then be connected to your Kubernetes cluster.

Deploying Locust to your Cluster

Now I’d recommend that you set up some environment variables which we will need so run each of these commands.

$ PROJECT=$(gcloud config get-value project)
$ REGION=europe-west1
$ ZONE=${REGION}-c
$ CLUSTER=distributed-locust-testing
$ TARGET=${PROJECT}.appspot.com
$ gcloud config set compute/region $REGION 
$ gcloud config set compute/zone $ZONE

We should also enable all of the necessary APIs using:

$ gcloud services enable \
    cloudbuild.googleapis.com \
    compute.googleapis.com \
    container.googleapis.com \
    containeranalysis.googleapis.com \
    containerregistry.googleapis.com

Next you need to run the following command in order to clone the files needed for the next steps.

$ git clone https://github.com/GoogleCloudPlatform/distributed-load-testing-using-kubernetes

If you list the files in your directory you’ll see that you now have some new ones there. Now we need to build a Docker image and push it to Google Container Registry. We can do this by running the following:

$ pushd distributed-load-testing-using-kubernetes/

$ gcloud builds submit --tag gcr.io/$PROJECT/locust-tasks:latest docker-image/.

This will take around a minute in order to complete each of the steps in the Dockerfile. You should be returned a message which looks like this if it has completed successfully:

CREATE_TIME: 2022–03–20T18:56:13+00:00
DURATION: 1M3S
SOURCE: gs://infra-agent-999999_cloudbuild/source/1647802572.889189–59f5e6f8582a45978bef34343012cbc4.tgz
IMAGES: gcr.io/infra-agent-999999/locust-tasks (+1 more)
STATUS: SUCCESS

Now we can deploy our sample web application to the cluster by running:

$ gcloud app deploy sample-webapp/app.yaml --project=$PROJECT

This will deploy a sample web application which was cloned from the GitHub repository to App Engine. This may take a couple of minutes.

Next we need to replace the [TARGET_HOST] and [PROJECT_ID] in our Kubernetes configurations. We will be replacing them with the environment variables that we set earlier. This is done by running:

$ sed -i -e “s/\[TARGET_HOST\]/$TARGET/g” kubernetes-config/locust-master-controller.yaml
$ sed -i -e “s/\[TARGET_HOST\]/$TARGET/g” kubernetes-config/locust-worker-controller.yaml
$ sed -i -e “s/\[PROJECT_ID\]/$PROJECT/g” kubernetes-config/locust-master-controller.yaml
$ sed -i -e “s/\[PROJECT_ID\]/$PROJECT/g” kubernetes-config/locust-worker-controller.yaml

After these have been replaced we can now deploy each of our components to the cluster. This consists of the Master deployment, the Worker deployment, and a Service. This is done by running:

$ kubectl apply -f kubernetes-config/locust-master-controller.yaml

$ kubectl apply -f kubernetes-config/locust-master-service.yaml

$ kubectl apply -f kubernetes-config/locust-worker-controller.yaml

Now if you take a look at your pods you will see that there is one Master and five Workers running in your cluster. You can see this by entering:

$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
locust-master-67bd6b669b-wlbc6   1/1     Running   0          20m
locust-worker-656b94465c-5s8cj   1/1     Running   0          20m
locust-worker-656b94465c-mc92z   1/1     Running   0          20m
locust-worker-656b94465c-qw6v6   1/1     Running   0          20m
locust-worker-656b94465c-thgjv   1/1     Running   0          20m
locust-worker-656b94465c-zxbsb   1/1     Running   0          20m

Congrats, you have successfully deployed Locust to your cluster!

Executing Distributed Load Tests

Now we’d like to start running some distributed load / performance tests against the sample web application which we deployed to the cluster earlier. First we need to get the External IP address of the Locust master and set it to an environment variable by running:

$ EXTERNAL_IP=$(kubectl get svc locust-master -o yaml | grep ip | awk -F”:” ‘{print $NF}’)
$ echo $EXTERNAL_IP

This will print the External IP to Cloud Shell. You should now be able to access the Locust Web UI by opening this address in your web browser:

http://$EXTERNAL_IP:8089
For example: http://34.140.153.84:8089

This should give you access to the Locust Web UI as seen in the screenshot below.

Locust Web UI

Next all you need to do is enter the number of users you would like to simulate and enter the hatch rate that you would like to use and you’re good to go! If you ever need to increase the number of Workers in order to simulate more users you can do this by scaling up your Worker deployment using this for example:

$ kubectl scale deployment/locust-worker --replicas=20

Important Notes

Please keep in mind that generating large amounts of traffic to external systems could be considered a Distributed Denial of Service (DDoS) attack. Make sure that you read over Google Cloud Platform’s Acceptable Usage Policy and Terms of Service to ensure you are not breaking it!

Also important to know that you can quickly run up quite a hefty bill if you continually run your Google Kubernetes Engine (GKE) cluster for long periods of time. Remember to terminate all resources used as part of this tutorial in order to avoid incurring any unexpected costs. The steps for doing this are below.

Delete your Google Kubernetes Engine (GKE) cluster
$ gcloud container clusters delete $CLUSTER --zone $ZONE

You should then also go to Google Container Registry in the console and delete the locust-tasks folder with the image we created earlier. That should cover everything!

References

Distributed Load Testing using GKE: This is Google Cloud’s GitHub repository for running these distributed load tests. Some of the steps and information in this guide is taken from this repository so please also use this as a base if you run into issues. I tried my best to create a more detailed guide here using the steps in this repository in order to make it easier to follow along.

If you have any questions feel free to comment here or message me on LinkedIn and I’ll try my best to help!