Implementing Shift-Left Performance Testing in Insider Infrastructure
Performance testing is an integral part of the development cycle at Insider. We serve customers globally and it is critical for us to ensure that our infrastructure is optimized for speed, scalability, and stability.
Currently, we have 2 stages to maintain our system’s performance, by observing the performance before and after the deployment. In the first stage, we are performing tests just before going to production on the pre-prod environment. If everything goes well and the new features are deployed in production, we utilize feature flags to monitor their effects. This approach allows us to identify and fix any unexpected issues before they impact a large customer base.
While the feature flag approach requires an additional manual monitoring effort after each deployment, it is a mandatory process for protecting our system performance. However, we have realized that testing the entire new development just before deploying often results in costly and time-consuming fixes if performance issues are identified at the end of development. Therefore, we’ve decided to shift our performance testing earlier in the development cycle. The concept of “Shift-Left Performance Testing” revolutionizes this approach by incorporating performance testing much earlier in the development process. This shift has significant benefits: it promotes early detection of issues, resulting in more efficient and cost-effective fixes. It encourages a proactive rather than reactive approach to performance.
Performance Testing Tools
Various open-source and commercial performance testing tools are available, the most popular are JMeter, K6, Locust, Ddosify, and Gatling. Each has its strengths and weaknesses. We previously used JMeter and Locust, but we found that JMeter struggled with the high-load generation, consuming excessive CPU and memory. Conversely, while Locust was more efficient, managing test scenarios in Python became burdensome as user flows frequently changed. This led us to explore new tools, guided by the following criteria:
Codeless Test Scenario Creation
Creating test scenarios requires implementing complex logic. We don’t want our developers spending too much time writing scripts for these tests.
UI Support for Test Result Analysis
Different teams need to work together, so it’s important for us to have a testing tool with a user-friendly interface to display test results. This way, we can easily share results between teams.
API integration
We want to integrate the performance testing process into our CI/CD pipeline. This way, we can test each pull request separately.
There are some other nice-to-have features we wished for a performance testing tool but these 3 ones were critical. As we mentioned above, we would like to try new tools except for JMeter and Locust. So the options are K6, Gatling, and Ddosify. We’ve examined these tools and identified some advantages and disadvantages.
K6
K6 offers open-source and Cloud options. Since it is a Grafana ecosystem tool, it has a relatively large community. The open-source version does not have UI support for both test scenario creation and test result reporting. The cloud version has UI support for reporting but it requires JavaScript to create complex test scenarios. We are heavily using JavaScript in our developments. However, for ease of maintenance, we are considering creating test scenarios within a No-code UI environment. Considering these factors, we concluded that K6 isn’t the best fit for our needs.
Gatling
Gatling shares many similarities with K6, particularly in terms of product deployments. Besides offering open-source and Cloud options, Gatling also provides a Self-Hosted version, which can be found in various Cloud provider marketplaces. It is written in Scala and it uses a domain-specific language (DSL) for test scenario creation.
We’ve found the Cloud marketplace deployments to be highly efficient, given they require almost no setup costs and are easily manageable. However, the use of DSL for the creation of test scenarios doesn’t align with our requirements. As a result, we’ve decided not to use Gatling.
Ddosify
Ddosify is a relatively new one, but it is growing very fast in GitHub. Its No-code, High-performance platform, ability to support Assertions, Correlation, Debugging, and Test Data import, and especially its Postman import, were all attractive features. Ddosify offered three separate products: Ddosify Engine (CLI), Ddosify Self-Hosted, and Ddosify Cloud. We will not deep dive into the details of these different deployments, the picture they are serving in their GitHub repository provides enough information. We preferred the cloud version because it requires no maintenance effort on our part, and it also offers a feature that allows for geographically targeted load testing.
Shift-Lefting Performance Testing
Our strategy involves testing each development change for performance. We began by creating test scenarios for the most frequently used user flows, integrating these scenarios into our CI/CD pipeline to run with every pull request.
We used Ddosify’s Postman Import feature to import all of our endpoint configurations, a time-saving measure as we already use Postman for API testing across all teams at Insider. This eliminated the need to duplicate configurations and saved enormous setup time for us. After that, we used Correlation to combine endpoints with each other to create user flows. Lastly, we used the Assertion feature to build success/fail logic for each scenario.
Shift-Lefting Performance Testing
Our strategy involves testing each development change for performance. We began by creating test scenarios for the most frequently used user flows, integrating these scenarios into our CI/CD pipeline to run with every pull request.
Since effortless creation of user scenarios was a key requirement. We used Ddosify’s Postman Import feature to import all of our endpoint configurations, a time-saving measure as we already use Postman for API testing across all teams at Insider. This eliminated the need to duplicate configurations and saved enormous setup time for us. After that, we used Correlation to combine endpoints with each other to create user flows. Lastly, we used the Assertion feature to build success/fail logic for each scenario.
Conclusion
To sum it up, our move to “Shift-Left Performance Testing” has changed the way we handle performance testing. Now we can spot issues sooner and we’ve built a forward-thinking culture around performance. Our decision to use Ddosify as our testing tool has further improved things. It saves us three times the amount of time we used to spend with other tools we’ve used before, thanks to its unique features and capabilities. It helps us develop reliable, adaptable, and performant systems for our customers globally.
If you enjoyed this, you might also like Deniz Parmaksız’s article entitled “Deploying AWS Lambda Functions for Machine Learning Workloads”.
