That is some serious Load!

If you are a software engineer you most probably have heard about load or stress or soak testing. In the this blog post we will introduce our unique GWI story of how we introduced a nice and easy solution to perform “shift-left load testing”.

But before starting our technical details and architecture let’s do a theory review….

Being a software or devops engineer means you are more or less aware that having unit tests in your code is important. You need them to ensure each code change or feature addition (a) doesn’t break the rest of the system and (b) works. But on top of that you need various kind of tests like integration tests, e2e/functional tests and of course performance tests. We will focus on the latter ones today.

Why do we need performance test I hear you ask! Because we need to evaluate how a system performs in terms of responsiveness and stability under a particular workload. Performance tests are typically executed to examine speed, robustness, reliability, and application size.

But which types of performance tests should you conduct, what’s the difference between load tests and stress tests, and which test is suitable for which situation?

Load Testing

Load test when you want to determine how many users your system can actually handle. You can configure tests to simulate various user scenarios which can focus on different parts of your system (such as a login page).

You can determine how the load behaves when coming from different geo-locations or how the load might build up, then level out to a sustained level.

Stress Testing

A stress test is a type of performance test that checks the upper limits of your system by testing it under extreme loads. Stress tests examine how the system behaves under intense loads and how it recovers when going back to normal usage. Are the KPIs like throughput and response time the same as before spike in load?

Stress testing can be conducted through load testing tools by defining a test case with a very high number of concurrent virtual users.

Soak Testing

While load testing is primarily concerned with performance assessment, and stress testing is concerned with system stability under extreme conditions, soak testing is concerned with reliability over a longer period of time.

A soak test uncovers performance and reliability issues stemming from a system being under pressure for an extended period.

K6 comes to play

k6.io is well known tool in the community and we decided to have a look on it as alternative to Apache JMeter or Gatling.

Lets summarise some of the features before describing how to get started!

• Developer friendly API and CLI
• Designed to automate load testing. You can add assertions and checks for your APIs
• Create load tests with Javascript or Typescript (we use the second!)
• Multiple storage capabilities. It comes out of the box with integration with multiple backends NewRelic, InfluxDB, Datadog, Kafka and many more.
• Monitoring through Grafana never been so easy
• Multiple extensions to give you more power on writing test suites
• High performance tool — k6 on top of Go

And let’s get started!!

We decided to use Typescript cause:

• Improves the ability to safely refactor your code
• Improves readability and maintainability
• Allows you to drop a lot of the defensive code previously needed to make sure consumers are calling functions properly
• Assists you with catching bugs on build time instead of runtime
• Enables a larger team produce production-ready code easier

To do so you can use the official repo as a template for your test project

To run tests we need first to install dependencies from package.json

$ yarn install

Then to build your code just execute

$ yarn build

Now all the tests compiled and can be run under /dist folder

$ k6 run dist/example.js

Now let’s review how a simple test can be organised to be executed with k6.io. First we need to define any needed model type that can be used to pass the payload on a given request.

We can create a model.ts file under src/lib to store different types of model entities that we need to pass in our tests for example:

The test script is actually organised in different parts.

First we need to define k6 options that describes the test run behaviour.

Of course we used only a subset of options from what k6 provides. For a complete list of options to configure your test script refer to k6-options.

Let’s see how a test script is defined with setup and tearDown hooks.

In the above example we created a set of users in our setup functions and delete them after the test is finished. With this example we can automate user authentication passing the needed token to the headers of our actual test requests. We somehow isolate our tests from the entire system under test.

Checks also validate boolean conditions in your test. Testers often use checks to validate that the system is responding with the expected content. These are important factors that we can make sure how many requests are not returning 200 OK thus validate how the system can perform under load. QA/Devs can actually experiment and add more validation points (checks).

Now let’s start by running our tests locally. There are multiple ways to do that either by using k6 native support or by using official docker image (we prefer the second!!)

For installing k6 packages locally refer to installation.

Now that we have our packages installed we will run our tests with different options to mimic behaviour of load, stress or peak test.

Simple k6 run command

$ k6 run — vus 10 — duration 30s dist/script.js

Smoke test

The minimum configurations contain 1 VU and running the test no more than 1 minute. If there are no errors produced you can continue to the next type.

Load Test

You should run a Load Test to:

1. Assess the current performance of your system under typical and peak load.
2. Make sure you continue to meet the performance standards as you make changes to your system (code and infrastructure).

Note, this test has one simple threshold. The response time for 99% requests must be below 1.5 seconds. Thresholds are a way of ensuring that your system is meeting the performance goals you set for it.

Soak Test

We recommend you to configure your soak test at about 80% capacity of your system. If your system can handle a maximum of 500 simultaneous users, you should configure your soak test to 400 VUs.

The duration of a soak test should be measured in hours. We recommend you to start with a 1 hour test, and once that one is successful, extend it to several hours. Some errors are related to time, and not to the total number of requests executed.

CI / CD Integration

For integrating our tests with our CI process we used GitHub actions as it can be easily triggered from our code base. The simplest workflow that is currently used from our teams is the own described below. Of course there are plenty of examples officially so you can get started.

We can provide the test script as input and the platform that we want to run our tests against (test, staging or even production) using github input events

${{ github.event.inputs.test }}

While seeing at the end of the workflow we used the upload-artifact action to upload the HTML report generated from the load test. This is uploaded directly to Github and has a configurable retention policy.

Grafana Integration

Grafana can make a difference when you want to visualise metrics in a historical way. Grafana is a modern monitoring platform that is used to get the correct data when you need it. Grafana can easily be installed and used with a predefined set of dashboards for your k6 test suite. We are going to describe more about the process of installing a local instance and running your tests against it.

As database for storing metrics we used InfluxDB that comes with predefined dashboards for Grafana. We will use docker-compose to install InfluxDB, Grafana and run our test suite against the system.

For the tests only need to define the environment aka InfluxDB output URL and the volume mapped to the test script folder (under dist after building our typescript files).

Running the above from CLI the tests will be executed and you can actually visualise the metrics after. Dashboard can be imported directly from k6-dashboard.

And the outcome is the following. Widgets are covering most of the metrics generated from k6. So adding Grafana in our stack we managed to get some shift-left observability integrated in our workflows.

Development is no longer a linear journey from point A to point B. As more projects shift into a state of organic growth, user feedback and constant experimentation are increasingly becoming the norm, if not the standard for engineering.

Conclusion

So to recap we experimented with k6.io and we concluded that we can maintain a test suite that can be developed from different engineering teams (QA/Devs/SDET). This is the main advantage against other tools in the market.

Challenges we faced were to map all the entities of different APIs to typescript objects (still in progress) in order to have a fully customisable environment.

The future is bright and k6 has a very strong community that can be used for support and collaboration in order to promote performance testing in your org.

Stay tuned as we will go deeper and review the solid capabilities of k6 in a series of posts from us soon.

Resources:

k6 Blog - Load & Performance testing Blog - Grafana k6

GitHub - grafana/k6-template-typescript: Template to use TypeScript with k6