How to take aim at less stressful performance tests with Gatling
Performance tests are a key source of information, useful to developers, business managers and stakeholders for a variety of reasons. I’m going to explain how our team has used the free, open-source tool Gatling to carry out performance tests that produce the most accurate results for APIs, while taking up a relatively small amount of resources.
A few words about performance testing…
Performance testing determines the capabilities of a system, by measuring the below items:
- Processing Time — The time taken by the server to process a request.
- Latency — The delay involved for a request to reach the server and back.
- Response Time — The total time it takes from when a user makes a request until they receive a response. The Response Time equals the Processing Time + Latency
- Throughput — Indicates the number of transactions per second an application can handle.
- Scalability — Measures an application’s ability to scale up or scale out, in terms of any of its non-functional capability.
These items feed into the following types of tests:
- Load testing — Tests the maximum amount of load that can be placed on a system or software.
- Stress testing — Tests the different failure and recovery points of a system to gauge its parameters for stable operation.
- Endurance testing — Tests the operation of a system under normal levels of load over a prolonged amount of time.
- Scalability testing — Helps to gauge the ability of a system to cope with higher levels of non-functional parameters, such as the volume of requests, data and users.
- Spike testing — It tests the behaviour of a system when it comes under sudden, large volumes of load.
- Volume testing — Tests the volume of data that can be processed by a system.
What can performance tests do for your business?
Load testing can monitor the system’s response times for each of the transactions during a set period of time. This type of monitoring can provide a lot of useful information, especially for business managers and stakeholders, who look for conclusions based on these results, along with any data to support these findings. Load testing can also bring attention to any problems in the application software, allowing engineers to fix these bottlenecks before they become more problematic.
The problem with traditional performance tests (or synchronous processing)
Most traditional performance testing tools work like this:
- 1 virtual user = 1 Thread
- 50 virtual users = 50 Threads
- 10,000 virtual users = 10,000 Threads
If you have 10,000 requests running simultaneously, most of the time your threads will be sleeping. There will probably be lots of context switching and lots of CPU utilisation will occur at the time of process scheduling. This approach consumes approximately 60% to 70% of CPU and memory at the time of execution, which degrades the system performance. Under this methodology, the user waits for the server response when sending a request. This is sometimes called Synchronous processing.
What synchronous processing looks like in action
Magic bullet: Why we chose Gatling
● We were after a performance testing tool that is built on the Scala language
● It enables us to performance test at an API level
● It can produce more accurate graphs
● It is free and open-source (developed in Java / Scala)
● The scripting language has its own DSL
● It works with any operating system and any browser
● It supports HTTP/S, JMS, and JDBC protocols
● It presents colourful reports in HTML
Performance tests that act in concert (asynchronous processing)
Gatling uses a more advanced engine based on Akka. Akka is a distributed framework based on the actor model and allows for fully asynchronous computing. Actors are small entities communicating with other actors through messaging, able to simulate multiple virtual users with a single thread. Gatling also makes use of Async HTTP Client.
Below is an example of the Akka distribution model:
Along with Akka, Gatling also uses the Java framework Netty, which I believe is the best non-blocking I/O (NIO) framework running on Java Virtual Machines (JVM). This approach increases the response time of the Threads and makes the application even more responsive, because of the following reasons:
- Blocking behavior of an application would mean that the threads of an application have to wait until the data is available in the stream, or the data is fully written in an application. This directly affects performance as threads become blocked.
- But NIO threads interact with channels and do not wait until data becomes available. This behaviour prevents the thread from being blocked, then once the data is available, it can be fetched.
Below is an illustration of the Netty framework:
With Gatling implemented, you can see the big efficiency savings it can make on your resources from the picture below:
How to configure your Gatling domain
We set up all the configuration items located in the “conf” directory:
logback.xml — This file allows you to configure the log level of Gatling.
gatling.conf — All configurable items will be present in this file. Initially all the config parameters should be in commented status. If we want to change the value of any given parameter, we can do that in gatling.conf. Normally we set up items such as Min Response Time and Max Response Time.
We keep all the Test Data files under this folder. Later, we supply this data into Scala Simulation DSL Script as per the requirement.
A simulation is a real Scala class containing 4 different parts. The simulator will run the Gatling scenario.
- The HTTP protocol configuration
- The headers definition
- The scenario definition
- The simulation definition
Request & Response Transformation
Once a scenario starts running, request and response transformations to and from server take place.
We can check a few things like HTTP status, any required labels and any data from the server response.
The Assertions API is used to verify that global statistics such as response time or the number of failed requests matches the expectations for a whole simulation.
setUp(scn).assertions(details(“Search” / “Index”).failedRequests.percent.is(0))
These are used to write the output content to Real time Graphs, Files and Console
How we tested the Travel Search API
This API allows customers to search for hotels, cruises, tours and holidays. We ramped up from 1 to 25 requests per second. We ran this test for the duration of 3 hours.
We had the following performance Service Level Agreement (SLA) criteria for the API:
The response time should be less than or equal to 0.15 secs.
The failed request percentage should be less than or equal to 2%.
The request per second should be greater than or equal to 12.
Why Gatling reports make for nice reading
Below is an example of the output reports from our tests against the Travel Search API, and highlight just why we think they make good, colourful reports with useful data. The following pictures illustrate the Global menu, which points to consolidated statistics. The Details menu points to per-request-type statistics.
Indicators at Global Level
Active users over time
Response time distribution
Response time percentiles over time
Requests per second over time
Responses per second over time
Indicators at Details Level
Response Time against Global RPS
Gatling Plugins For Jenkins for continuous integration
This recommended plugin allows you to:
- Keep track of a Gatling simulation, providing performance trends across builds.
- Publish detailed reports for each build.
How to install
Launch the Jenkins home screen, Navigate to Manage Jenkins -> Manage Plugins screen and install the Gatling Plugin.
Configuring the Gatling plugin for Maven for your project to execute Gatling simulations is described on this page. Please follow these instructions to configure it.
Using the Gatling plugin
As soon as you’ve properly configured your job and launched a build, you’ll see two changes on your project dashboard:
- A new entry will be available in the left summary: Gatling.
- A graph, displaying the mean response time trend of your last 15 builds, will appear. (please refer to the below screenshot)
You can find the full gatling-jenkins plugin configuration and usage from this link.
The API’s we test with Gatling
To date we have done performance testing of Google AMP API, Travel Holiday API, Travel Places and Travel Search API.
For more information