Step by step guide to chaos testing using Litmus Chaos toolkit

by Sunit Parekh and Prashanth Ramakrishnan

In this article we will describe how to perform chaos testing using Litmus chaos toolkit, a popular chaos testing tool.

There are 4 major steps for running any chaos test.

  1. The first step is defining a steady state, which means defining how an ideal system would look like. For a web application, the home page is returning a success response, for a web service this would mean that it is healthy or it is returning a success for the health endpoint.
Chaos Testing steps

Lets begin,

Step 0: Kubernetes Cluster with Bookinfo Application & Monitoring in place

To learn more about chaos testing, first we need to have an application under test, for this demo, we are going to have a BookInfo application deployed on a single node Kubernetes cluster. Along with it, we have Prometheus, Grafana, Jaeger & Kiali setup with Istio service mesh.

1 ) Setup Kubernetes Cluster: Get your Kubernetes cluster up and running with Docker1 as container runtime. To keep it simple install Docker for Desktop and also start Kubernetes along with it. However, you can also use Minikube, k3d, kind to set up local k8s clusters.

2) Setup Monitoring: Next to setup Istio along with all monitoring tools such as Prometheus, Grafana, Jaeger & Kiali

Install Istio:

$istioctl install — set profile=demo -y

Install monitoring:

## Change ISTIO_RELEASE_URL to specific istio release version 
$kubectl apply -f $ISTIO_RELEASE_URL/samples/addons/jaeger.yaml
$kubectl apply -f $ISTIO_RELEASE_URL/samples/addons/prometheus.yaml
$kubectl apply -f $ISTIO_RELEASE_URL/samples/addons/grafana.yaml
$kubectl apply -f $ISTIO_RELEASE_URL/samples/addons/kiali.yaml
Istio and monitoring pods installed in istio-system namespace of k8s custer

3) Install Bookinfo application with Istio service mesh enabled and envoy sidecar installed.

Bookinfo application overview with 4 microservices and sidecar proxy injected
Download BookInfo yaml from Istio website:

Install Bookinfo with envoy proxy injected as sidecar container into default namespace:

$istioctl kube-inject -f book-info.yaml | kubectl apply -f -
Bookinfo application pods running in default namespace

4) Verify applications are running and envoy proxy deployed.

Do port forwarding for the productpage service and check in your browser.

$kubectl port-forward service/productpage 9080:9080

If you open localhost:9080 in your web browser, you should see something like this

Product page of Bookinfo application

So now I have my application up and running inside the k8s cluster.

Step 1: Define steady state

Steady state for the bookinfo application is that the product page should keep rendering without any issues. Means http://localhost:9080/productpage?u=normal should return 200 http status code under continuous load.

To check my steady state condition let me first generate continuous load on my bookinfo application using a command line tool called hey and monitor it.

$hey -c 2 -z 200s http://localhost:9080/productpage

Above command generates continuous load on the product page for 200 seconds with 2 concurrent workers.

Here is a quick view of the Kiali dashboard showing all pods healthy and review service responding in a 100–200 ms timeframe, which internally calling rating service responding in avg 50–60 ms.

Bookinfo application — Kiali Dashboard

Step 2: Introduce chaos

All set, now time to introduce chaos in the system. Let’s first understand Litmus core concepts before we jump into execution.

1) Install Litmus: First step is to install an operator (Litmus Operator) into the Kubernetes cluster where we like to introduce chaos. Limus operator adds 3 custom resource definitions related to Litmus chaos into k8s cluster. You can also use Helm charts to install Litmus operators and its web UI. However, for simplicity I am going with direct yaml and install only operator.

$kubectl apply -f
List of all CRDs added as part of installing Litmus operator

2) Setup Experiment: After that we need to add a specific experiment in the namespace where we like to introduce chaos. List of all the available chaos are listed here. Lets add a network deploy chaos experiment into the default namespace where we have our bookinfo application installed.

$kubectl apply -f
pod-network-latency experiment added in default namespace

3) Apply Permissions: Now we need to give permission using RBAC to allow chaos experiments to run.

$kubectl apply -f
pod-network-latency-sa added in default namespace

4) Run Chaos: Using ChaosEngine custom resource definition, we inject network delay chaos. Please look at the following yaml (network-delay-engine.yaml) of kind ChaosEngine for introducing network delay of 2 sec for ratings deployment for about 100 seconds affecting all pods under deployment. Delay in ratings service response is going to indirectly delay review services and which indirectly adds delay to product page.


Please check comments in above yaml to learn about different configurations. Details about each configuration can be found in documentation provided by Litmus toolkit here.

$kubectl apply -f network-delay-engine.yaml

Here is pod watch of default namespace and notice bookinfo-network-delay-runner , pod-network-latency-rp2aly-vg4xt and pod-network-latecy-helper-hhpofr pods doing the job of introducing network delay in rating service.

pods status during chaos testing from start to end

5) Observe Result: Use Kubernetes describe command to see output of the chaos run we had in previous steps. Lets first notice increased time in review service response time on Kiali.

Kiali dashboard showing 2s+ response time for Reviews service

Now let’s describe ChaosEngine and ChaosResult to see the result in Litmus custom objects description.

Chaos custom resource lookups
Events stream using describe from chaosengine custom resource bookinfo-network-delay
Events stream using describe from chaosresult custom resource bookinfo-network-delay-pod-network-latency

Lets increase delay to 6 seconds (6000 ms) and re-run the chaos

Change network-delay-engine.yaml with config NETWORK_LATENCY set to 6000 ms.

Reviews services turning red when 6s delay introduced by ratings service
Product page loading with error handled gracefully now showing ratings information

Step 3: Verify steady state

During the chaos test time we continuously accessed the system and observed 200 responses for the product page.

hey command output showing result of productpage response as all 200 status code

Observed 2 sec delay in response time on review service on Kiali dashboard.

Kiali dashboad with 2s delay on Reviews service

Step 4: Rollback chaos

In our case of network delay, since the chaos duration was set to 100 sec. It stopped automatically after 100 sec. So nothing to be done. Just observe that our system is back to normal.

On Kiali dashboard we see returning it to normal with review response time less than 100 ms timeframe and rating response time in 50–60 ms timeframe.

Kaili dashboard with Review services responses in double digit ms time (all back to normal)

Q & A

Can I use the Litmus tool with any other container runtime like containerd?

Yes, Steps in this article are keeping Docker as container runtime, however, if you have other runtimes like containerd, please read configuration on Chaos website for different configurations needed to run chaos experiments.

Where can I find a list of all chaos experiments available?

Litmus has some predefined chaos experiments available which can be found here, but it does not limit us to define our own experiments and run them in our own environments.

How to debug issues if any?

While running chaos using ChoseEngine CRD, use following flag jobCleanUpPolicy: ‘retain’ to keep pods in complete state (and not to be deleted after chaos run) which provides ability to look at logs of the pods.


Above commands and code checked into the public repository on Github.

Watch all of above in action in XConf 2021 online conference talk

Learn more with hands-on tutorial on Litmus site

Online hands-on with Litmus tool on KataCode

 by the author.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store