Load and performance test in Getir

Touching millions of users requires a lot of care. Your services should be ready for users in every scenario. As Getir testers, we have been dealing with huge amounts of load on our services, so we needed to try a number of load testing tools and frameworks. We created various load test scenarios and cases to understand which is the best tool for us and see if the requirements match our limits.

• Jmeter
• Locust
• Gatling
• Webload

Considering our needs, the preference for open source and taking scalability into account, we have chosen Locust. Let’s explore the details of our selection and why most of the testers choose Locust.

Why did we choose Locust?

Jmeter is based on JVM so it makes things slower. Locust is an open source framework that is developed in Python. In addition to the test as a code feature, it’s highly scalable as it’s a purely event-driven application. For this reason, there is a large audience that prefers Locust over JMeter. We have determined some important features of the tools we used.

Check it out:

We realized that if we run the same script with the same scenario and same duration, Locust is able to make more requests and the CPU usage is almost half as much. If we want to reach the same RPS with the Jmeter, we need to use more CPU, so our cost increased a lot when it was desired to run these tests in the Cloud.

Jmeter has a structure that assigns one thread for each user activity. If we try to run the script on a standard machine and we observed that its RPS amount does not meet Getir’s needs. Especially in GUI mode, it gets an “OutOfMemoryError” error. It’s a bit more effective in non-GUI mode and we can probably solve this memory error by optimizing java heap size, but is this really necessary? Particularly when we already have a better tool that is well optimized. Yes, Locust has a different perspective about user simulation which uses an asynchronous thread approach. It allows to easily simulate simultaneous users on a single machine while performing complex tests with many steps in it. We have observed that this approach allows us to ensure more users with the same conditions in our tests.

So, how do we do that?

• First of all, we have a kick-off meeting to determine the requirements with the whole team. We discuss the user scenario and measure the weights of each endpoint according to the old data. By the end, we know how much load we need to supply for each service.
• Then we develop the test script and make a cold shot.
• We are deploying our load test script to an EC2 service that has auto-scale workers to the needs of RPS amount (Locust has already supplied auto-scale).
• We prepare a pre-prod env that is similar to the production environment, ensuring that it’s the same data and same env with the same numbers of pods.
• Then, we run the test script.
• We monitor the services to see where our bottlenecks are.
• Finally, we report back to the team.

Thanks for reading and feel free to reach out :)

Cheers🍻