Performance Testing in HafidzIsMe API

There’s more than just Unit Testing.

This article is written as a part of Individual Review competency for Software Projects course 2020 at Faculty of Computer Science, University of Indonesia.

Courtesy of https://blog.testlodge.com/performance-testing/

Introduction

During the development phase, the back-end API of HafidzIsMe had been going through many extensive unit tests. Besides complying to Test-Driven Development paradigm, they help us identify potential edge cases and verify business logics, thus minimizing bugs that could arise during production. However, such tests are not sufficient to ensure the overall quality of our application, especially in term of perfomance. Hence, we need more testing strategies that could determine the quality of our application in other aspects. This article will discuss performance testing and its application to HafidzIsMe back-end API.

Non-Functional Testing

There are different fields of software testing, with Unit Testing being categorized as functional testing; that is, a type of testing which verifies the requirement specifications. On the other hand, there exists another type of testing: non-functional testing, which addresses non-functional aspects of the software. Such non-functional aspects could be performance, usability, reliability, or even security.

Performance Testing

A sub-type of non-functional testing is performance testing, which its main goal is to ensure that the tested software will perform well under their expected workload. In its own, performance testing consists of the following testing types:

1. Load Testing — conducted to determine how an application performs based on a certain number of users. Usually, a load test will increase the number of requests during the duration of the test, when given an expected workload (i.e. expected number of users or requests according to the design).
2. Stress Testing — conducted to determine how an application performs under a maximum, or over maximum number of requests. It is similar to load testing except the number of requests given. In stress testing, the goal is to verify that the application can continue to perform well under the designed maximum capacity. Different that load testing, it also used to determine at what point a system breaks down.
3. Endurance Testing — conducted to determine how an application performs over a prolonged period of time in a typical expected workload continuously.

Application of Performance Testing to HafidzIsMe API Endpoint

At this point, we know what performance testing is along with its sub-types of tests. With this in mind, we can demonstrate performance testing to the back-end API of HafidzIsMe, specifically to one of its endpoints.

The Endpoint

For demonstration purpose, we will use an endpoint that will retrieve the information of latest opened period in HafidzIsMe program: /api/tahfidz/periods/latest/. It is a read-only endpoint and does not require authentication, so the test will be easy to set up.

The Tool

There are many great tools are available for conducting performance testing, especially load testing, one of it being Locust, an open source load testing tool. Cited from its website, it can be used to define user behavior with Python code, and swarm the application/system with millions of simultaneous users. It is relatively easy to set up and does not require us to modify our application code. It also has a nice and informative user interface that informs the testing results, more on this below. Although Locust is mainly intended for load testing, it can also be used for stress and endurance testing.

Locust website

Installation and Set Up

Locust is available on PyPI and can be installed with pip through this following command:

pip install locustio

To set up a testing scenario with Locust, we need to create a Python file that defines the user behaviours. We can name this file with anything we like but for this demonstration, the file will be named locustfile.py .

The below snippet is a simple testing scenario to the /api/tahfidz/periods/latest/ endpoint.

The code above defines the user behavior as a Python function that is decorated with @task decorator, inside APIUserBehaviour class, which inherits the TaskSet class from Locust library. We also need to define a class which inherits HttpLocust class in order to tell which user behavior class to be used (in this case, the APIUserBehaviour ).

Since HttpLocust provides access to a HTTP client, we can perform typical HTTP request to the URL/endpoint we would like to test. In this case, the endpoint being tested is wrapped into latest_open_period function.

Running Locust

To run Locust, simply open a command line terminal at where the locustfile.py is located, and run this command:

locust

By default, Locust will create a web interface at port 8089, we can access it from http://localhost:8089 .

Greetings by Locust

We can see three form fields, the first two fields will be used to determine how many users and users spawn rate for our testing scenario. The third field determines the host of the tested application. In this case, we will use http://localhost:8000 .

Load Testing

To perform a load testing, we will use a total number of users of 100 and hatch rate of 5. The numbers come from a rough estimation of a typical workload in HafidzIsMe website. The test will be run in roughly 3 minutes.

Load testing scenario

Below are the results of the test:

Summary of load test

The screenshot above shows the overall statistics of the test. We can see that a total of 9973 requests were made during the test execution. Fortunately there were no failures. We can safely say that with 100 concurrent users of 5 hatch size, the application performed well. Let’s take a look at the charts in detail.

• RPS over time graph

Requests per second

• Response times (ms) over time graph

Response times (ms)

• Number of users over time graph

Number of users

Stress Testing

For stress testing, the total number of users will be increased at a whopping 1000 users and hatch rate of 100. The test will be run in roughly 3 minutes.

Stress testing scenario

Below are the results:

Summary of stress test

We can see that a total of 15814 requests were made during the test execution. Just like the load test result, there were no failures. We can safely say that with 1000 concurrent users of 100 hatch size, the application performed well.

• RPS over time graph

Requests per second

• Response times (ms) over time

Response times (ms)

• Number of users over time

Number of users

Taking it Further…

Since the last stress testing didn’t give us any point of which the application “breaks”, we are going to amplify the number of total users up to a whopping 5000 users. The hatch rate will be still the same. Similar to previous tests, it will be run in roughly 3 minutes.

A more “stressful” testing scenario

Below are the test results:

Failures started emerging

As we can see from the statistics above, this time failures happened. At this point, we can say that the application had trouble in handling 5000 concurrent users. Detail of failures will be given later.

• RPS over time graph

Requests per second

• Response times (ms) graph

Response times (ms)

• Number of users graph

Number of users

• Failures detail

Detail of failures

All the 9 failures are ConnectionError , which were probably caused by the application’s inability to handle network congestion, hence 9 requests were dropped forcibly.

Endurance Testing

For endurance test, the idea is that we will run a test with a significant workload in a prolonged period of time. We will use a number of total users of 500 with a hatch rate of 100. The test will be run in roughly 15 minutes.

Endurance test scenario

Below are the results:

Summary of endurance test

Judging by the statistics above, the application handled concurrent 500 users in 15 minutes well enough. There were no errors which means that the endpoint being tested is stable in the long run.

• RPS over time graph

Requests per second

• Response times (ms) graph

Response times (ms)

• Number of users graph

Number of users

Next Steps

We have demonstrated performance testing scenarios to the selected endpoint and identified the point at which the application breaks. However, we could do more to improve the testing strategy. Perhaps we could include more complex testing scenarios. For example: calling more than one endpoint, calling endpoints in a certain scenario (login and access authorized endpoints), etc. Good testing strategies will more likely help us find which parts with poor or below expected performance.

Conclusion

Unit testings are not enough to determine the overall quality of our software. To determine the performance of the application on a specific workload, we can conduct performance tests. We can use Locust to perform kinds of performance tests. With its intuitive user interface and reports, Locust provides an easy way to perform performance tests.

This wraps up the article. If you have any suggestions, don’t hesitate to comment :)

Thanks for reading!

References

Web service performance testing - tips and tools for getting started

Locust - A modern load testing framework

Locust Documentation - Locust 0.14.6 documentation