Distributed load testing with Kubernetes and Predator

Image for post
Image for post

In this tutorial, we are going to use an open-source performance platform designed for APIs crafted at ZooZ (a PayU company) called ‘Predator’.
Predator is a distributed performance testing platform that uses Kubernetes abilities to schedule a distributed load on your APIs which is aggregated to one report in real-time.


While Predator supports more platforms like Docker engine and DC/OS we are going to focus on Kubernetes.
The simplest way to deploy Predator is by using the official helm chart.
Predator uses persistence to save all tests, reports and scheduled runs.
You can take advantage of Kubernetes Persistent storage or connect it to other data sources like Cassandra, PostgreSQL, MySql, and SQL Server.
For simplicity let’s go with persistence storage.

helm repo add zooz https://zooz.github.io/helm/helm install medium-tutorial zooz/predator --set persistence.enabled=true

If everything went well you should get the following output:

1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods -l "app=medium-tutorial-predator,release=medium-tutorial" -o jsonpath="{.items[0].metadata.name}")
echo "Visit to use your application"kubectl port-forward $POD_NAME 8080:80

After following the above instructions (port-forwarding) or using any other method you use to access Kubernetes services you should be in the Predator UI.

Image for post
Image for post

Writing the first load test

We are going to create our first API test using the Create test form.
While Predator supports composing complex load tests with weights between multiple scenarios, custom javascript injection, before section, capturing values from the responses and more (you can find all about this in the docs), we are going to write a one-step test that load tests a bin details service (a service which receives the first 6 digits of a credit card and returns info about the vendor, country, etc..)

To do so just go to the Tests section and press the Create Test button, use the form to configure your load test.

Image for post
Image for post

Run it

This is when the fun begins, Predator can use your cluster capacity to create tremendous load and aggregate all of it to one report in real-time by clicking one button.
To run a test just press the Run button under the Run Test column and configure your parameters.

Image for post
Image for post

In the above setup we requested Predator to run with 10,000 arrival rates meaning firing 10,000 scenarios each second (currently each scenario includes only one HTTP call), we also told Predator to use a parallelism level of 8 meaning to split the load between 8 pods and to run the test for 5 minutes.

Let’s check what is going on in the cluster using kubectl.

kubectl get pods | grep "predator"medium-tutorial-predator-5c78cbd96c-sn8pp                         2/2     Running   0          37mpredator.cfe37ec7-59c2-498c-bc5e-acb8cc07333f-157928372510274w4   2/2     Running   0          3m45spredator.cfe37ec7-59c2-498c-bc5e-acb8cc07333f-1579283725105dps9   2/2     Running   0          3m45spredator.cfe37ec7-59c2-498c-bc5e-acb8cc07333f-1579283725107fmqt   2/2     Running   0          3m45spredator.cfe37ec7-59c2-498c-bc5e-acb8cc07333f-1579283725108xdtf   2/2     Running   0          3m45spredator.cfe37ec7-59c2-498c-bc5e-acb8cc07333f-1579283725109k6rb   2/2     Running   0          3m45spredator.cfe37ec7-59c2-498c-bc5e-acb8cc07333f-157928372510c4w6g   2/2     Running   0          3m45spredator.cfe37ec7-59c2-498c-bc5e-acb8cc07333f-157928372510kx96k   2/2     Running   0          3m45spredator.cfe37ec7-59c2-498c-bc5e-acb8cc07333f-157928372510v5qph   2/2     Running   0          3m45s

As you may see Predator started a job that deployed 8 different pods to create a distributed load.

Inspect results

all the running pods are reporting to Predator about their results and predator aggregates everything in real-time to one visual report.

Image for post
Image for post
Image for post
Image for post
A real-time report showing latency, RPS and status codes

Predator also supports publishing the results to Prometheus or Influx by using push-gateway and viewing more detailed reports in Grafana.


In this short tutorial, we went over the basics of Predator.
you can find much more information regarding its functionality and features that are not covered in the docs.

Image for post
Image for post

Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store