Performance Testing using Artillery

At Till, we use Apollo Federated GraphQL for our organisation api graph which provides a gateway to connect various microservices.

SRE (Site Reliability Engineering) is very important to us as our systems serve thousands of customers across the globe.

When we talk about SRE, SLI (Service Level Indicator), SLO (Service Level Objective)and SLA’s (Service Level Agreement) become imminent.

The most important SLO’s for all services are to do with availability and performance.

We want 99% of our services to return a response in less than 500ms.

We want 99% of our services to return a non 500 error code.

For us to have the SLO’s setup, we needed a way to load test our apis to mimic production scenarios and measure our latencies and availabilities. This is where we came across Artillery.

We will talk about how to setup Artillery and run your first load tests in this article based on our learnings.

Ramping up to 5 users in 5 seconds (Time in x axis)

Outline:
1. Prerequisites
2. Installation of Artillery tool
3. Scripts writing using YML or YAML file
4. Report generation (HTML and JSON)
5. Point to take care while writing scripts

Prerequisites:

1. Node needs to be installed on your system, check it using the command “node — version” and refer website for node installation “https://nodejs.org/en/”

2. VS Code: We are going to use VS Code as an IDE https://code.visualstudio.com/download

3. Install YAML Sort extension for formatting yml file from VS Code marketplace.

Installation of Artillery tool:

Open VSCode terminal and type npm init and follow steps provided by terminal to generate package.json file

If you want to install artillery globally on the system then type at terminal

npm install -g artillery

or if want it to be only installed locally as a dev dependency for package.json then use

npm install artillery --save-dev

We used Faker library to generate random data for our API requests which can be installed using the below command

npm install faker --save-dev

Once installed, your package.json should look like below under dependencies and dev dependencies section

dependencies

Test scripts

For our purpose, we created a data folder containing two CSV files (staging_login.csv & production_login.csv) to store username and password separated by a comma which would be used in a generating a token based on the env the test would run in.

You can use different CSV files based on the type of data which can be referenced from the test file as below.

Pass data from CSV to the test using the payload section

Creating test data

We created request.js file inside data folder to generate random data using faker library.

Writing test scenarios

We created all our tests under tests folder in the project directory.
Artillery supports yml to write our test scenarios.

Config section is used for configuring settings such as the env, data file, load progression, plugins, etc.

Before section is used for running prerequisites required before the actual scenario.

Scenarios are the test cases which create requests to run the performance tests.

The project structure looks like the below:

project structure

Running the tests

Open terminal & run the command as below by passing in attributes such as env using -e.

artillery run tests/tests.yml -e staging

Test results are printed as below once executed:

terminal report

It’s always a good practise to create a separate config file and place all config section code in config.yml file to reuse it across multiple test files and then you can run the tests as below

artillery -c config.yml run tests/tests.yml

config.yml

Test Reports

Create a folder ex: reports to output the test reports.

We can generate both HTML and JSON reports.

Generating HTML reports is a multi step approach.

First, we need to generate a JSON report

artillery -c config.yml run tests/tests.yml -e staging -o reports/tests.json

Then using a JSON report, we can generate an HTML report.

artillery report — output reports/tests.html reports/tests.json

HTML Report

Few things to note

Use YAML Sort for formatting the code
Press Ctrl +P, it will open Quick Search. type “>F” and Select Format YAML

Quick Search

Unable to run multiple YML files
You cannot execute multiple YML scripts by passing folder names or by passing multiple YML files as it’s not supported by artillery and it’s an open issue at the time of writing this article.

Below commands is what not yet supported by Artillery to run multiple tests at once

artillery -c config.yml run tests/*.yml 
artillery -c config.yml run test1.yml, test2.yml

Follow Hierarchy
All Script sections should follow the same hierarchy and on the same vertical line in the yml file, or else some of the scripts will not work like expect or capture section of the yml file.

Use JSON.stringify() to get context, requestParams and response data
To get the data available at function parameter use JSON.stringify(context).
The context contains all npm, environment URL, and variables information

References :
https://github.com/artilleryio/artillery/tree/master/test/core/scripts

https://artillery.io/docs/guides/overview/welcome.html