Performance Testing with Gatling

When delivering software under tight deadlines, it’s very easy to treat performance testing as an afterthought, or even completely forget about it. However, as N26 is entering a period of immense international expansion, it’s important to be prepared for the increased traffic that is going to hit us. More importantly, we need to understand, estimate and analyse the level of traffic that we’ll have to expect, and how efficiently we will be able to serve our growing customer base without any service dropouts. If we hit peak loads, how are we going to gracefully handle any additional load?

These concerns can be answered by determining the capability of our services and APIs. Performance testing serves exactly this purpose.

There are different types of performance testing, which can be employed depending on the projected business need:

• Load testing — checks the application’s ability to perform under-anticipated user loads. The objective is to identify performance bottlenecks.
• Stress testing — involves testing an application under extreme workloads to see how it handles high traffic or data processing. The objective is to identify the breaking point of an application.
• Endurance testing — done to make sure the software can handle the expected load over a long period of time.
• Spike testing — tests the software’s reaction to sudden large spikes in the generated load.
• Volume testing — the objective is to check the application’s performance under varying database volumes.
• Scalability testing — the objective is to determine the effectiveness in “scaling up” to support an increase in the load. It helps to plan capacity additions to the existing system.

API Performance Testing

Software performance can be measured at different levels, e.g. the application or the API level. In this post, we’ll discuss performance testing at the API level.

APIs are backbone of a microservice architecture. The performance of an API is a critical factor: it doesn’t matter how extremely well built our front-end applications are, their performance depends on the one of the APIs. If an API takes several seconds to respond, the user experience of the application is badly impacted.

The following parameters help us to understand the performance of our API:

• Processing Time
• Latency
• Response Time
• Throughput

There are many tools on the market to measure performance at the API level. Traditional tools like JMeter use one thread per virtual user approach to generate loads. If we need to simulate 1k users, then Jmeter runs 1k threads, which results in excessive resource utilisation and degrading system performance. Other, more modern tools include K6, Artillery and Gatling. Gatling is one of the most widely known tools, which is also used by the QA team at N26. Let’s see how it works!

How Gatling works

Gatling is an open source load testing framework based on Scala, Akka (the implementation of the Actor Model on the JVM) & Netty (an asynchronous event-driven network application framework). It’s a highly capable and feature-rich tool.

Let’s take a scenario of testing the Certification API. In this scenario, we combine two requests:

1. POST to create certification
2. PUT to confirm the created certification

In addition to this, we need a number of virtual users, duration and acceptance criteria for the scenario.

• Ramp up from 1 to 25 requests per second for the duration of 30 minutes with the following performance Service Level Agreement (SLA) for the API:
• The response time should be less than or equal to 0.15 secs.
• The failed request percentage should be less than or equal to 1%.
• The request per second count should be greater than or equal to 12.

This scenario can be simulated with the help of assertions based on our acceptance criteria or SLA. We can define as many assertions as we need. Gatling provides a dedicated DSL for chaining the following steps :

1. Defining the scope of the assertion
2. Selecting the statistic
3. Selecting the metric
4. Defining the condition

All assertions are evaluated after running the simulation. If at least one assertion fails, the simulation fails.

For this scenario, sample statistics are generated as follows:

Gatling also generates an HTML report with a graphical representation of all the stats. A sample report can be found here.

Real-Time Monitoring

From the performance testing perspective, data is extremely important: real-time monitoring & metrics are needed to get continuous feedback, analyse the results and improve the performance. Gatling provides live metrics via the Graphite protocol, which can be persisted and visualised. We can configure a time series database like Influxdb as the data source and provide live monitoring with graphical tools like Grafana.

Curious to know what it looks like? Here is a preview of what we are going to achieve.

Real Time Monitoring Dashboard

Final Thoughts: establishing the baseline before testing

Load testing is at its most useful if the workload that is being tested and the workload that our API will be handling in real life are as similar as possible. It’s not very useful to know that our API can keep up with 500 requests per second if we don’t know whether our real traffic will be higher or lower than that, or whether the load will be uniform across all endpoints.

Because of this, we should start planning our tests by gathering usage data for our API. Before starting performance testing, we should have real-life data for:

• Average throughput in requests per second
• Peak throughput
• Throughput distribution by API endpoint
• Throughput distribution by users

Another key thing to think about is what the traffic we are going to simulate during the test will look like. The main options here are:

1. Repetitive load generation
2. Simulated traffic patterns
3. Real traffic

As always, it’s best to start with the simplest approach and evolve to more realistic options progressively. Using real traffic is an ideal scenario, although one that is not always feasible. Studying our traffic analytics and doing a simple probabilistic simulation is the best approach to have a near real traffic simulation: it will be close enough to the real thing to highlight any problems that we might have in production.

This is an overview of performance testing in a nutshell. Of course, as we are progressing on our journey to achieve a better and better fit to real-life scenarios, no doubt we will come across many challenges and new insights.

Happy Testing!

Interested in joining one of our teams as we grow?

If you’d like to join us on the journey of building the mobile bank the world loves to use, have a look at some of the Tech roles we’re looking for here.