Microservices Testing Strategies — Part 3
In the second article on microservices testing strategies, I shared the pre-deployment tests, what they are, how we are performing them at the Emirates Group, and how we are positioning them in PR pipelines.
This article is the last part of the microservices testing strategies. We will continue with post-deployment tests, test data management, pre-release testing approach, and other technical testing types which we are performing at the Emirates Group.
System Integration Tests (E2E Tests)
System Integration Tests (SIT) ensure we build the right system and test how the application behaves in an integrated environment. We test the important user flows from start to finish to ensure their behaviours are expected. We should not use any mocks or stubs for system integration tests. All system components should be integrated. This way, we can ensure that system-level integration is working as expected. These tests take more time than others. Therefore, we need to test the critical business flows at the SIT level.
For these tests, we follow the below procedure:
- Prioritize the backlog.
- Gather high-level business rules.
- Elaborate on those business rules and create the steps for them.
- Prepare required test data, endpoints, and expected results.
- Start to write test scenarios based on the above inputs.
After exploration and discovery, we create our test scenarios and implement SIT tests. You can also use cucumber or any other BDD library to add a Gherkin flavor to your tests or create your step classes to hide the details inside its methods. We prefer not to add one more layer with Cucumber and use vanilla JAVA to implement the tests with Gherkin fashion.
A sample SIT test case scenario is shown in the code snippet below.
A typical SIT pipeline is shown below.
After system integration tests are passed, we run the service-level performance testing. In microservices, we can do performance testing in several ways. First, we mock all external dependencies and hit the service endpoint directly to measure standalone service performance. This test gives us an isolated service performance.
We use Gatling as a performance tool and write the performance tests inside the microservice repository. This way, we keep the implementations for performance, component, and system integration tests inside the microservice repository.
In addition, we check the memory and garbage collector behavior using the Application Performance Monitoring (APM) tool and containers scale-up and scale-down behavior, CPU, and RAM usage using Splunk. These tools provide extra details for the service and the system-level performance, and it is really cool to use them along with Gatling performance test results.
Aside from the service level, we conduct performance tests by hitting the service over the API Gateway’s endpoint. This time we add the gateway factors to our performance tests. In real life, all channel requests pass through the gateway before reaching the services. That’s why it is also important to do performance testing over the gateway. Here, we may not use mocks or stubs. That’s why we pay attention to test data very carefully before starting performance tests. If the external systems are slow, it affects our performance test results, and by using APM tools, we figure out the problematic areas.
We placed the performance test step in the staging pipeline shown below.
For hybrid server architecture, we do performance testing for each if some of the servers are on-premise and some are in the cloud (AWS, Azure, etc.). This way, we can assess on-premise and cloud performance behaviors.
Also, in a hybrid architecture, data replications happen between on-premise and cloud servers. Therefore, we need to do XDCR (Cross Data Center Replication) tests to determine the maximum data replication performance. For example, we create a shopping cart on-premise, read it in the cloud server, update it on-premise, delete a product in the cloud server, and read it on-premise again. We create many scenarios about XDCR testing, and all these scenarios should run flawlessly with given pause intervals between each operation.
We conduct extra performance tests to evaluate system performance and stability. We perform spike testing by applying sudden loads and checking the auto-scaling, system behavior, and stress testing to see peak performance and load levels. On top of these, we perform endurance testing for an extended period to check the stability and resiliency of the service over a long period. Moreover, we check load-balancing factors in performance testing. We go through the load balancer, API gateway, and services in these tests. In these ways, we try to cover many aspects of performance testing for our microservices.
Other Technical Tests
Chaos testing focuses on the resiliency and integrity of the system by proactively simulating failures through a simulated crisis. This way, we know the system behavior when unplanned and random disruptions happen. These disruptions can be technical, natural, or disasters like an earthquake affecting the servers.
We are planning to use chaos testing tools such as chaos monkey, which randomly terminates virtual machine instances and containers inside your production environment. In this way, we test the system’s resiliency.
Mutation testing is another technique in which we inject mutants into our codes and check whether they are alive after the test execution. If these mutants are demolished, it is a good sign that our tests detect the failures; if not, the test code or test data should be improved. Mutation testing ensures the quality of our tests, not the service code itself, and they should be performed as early as possible in the PR pipelines.
Security and Vulnerability Tests and Scans
We added automated vulnerability and security scans in the service pipeline using Zed Attack Proxy. We scan our service with OWASP security and vulnerability rules in each new PR in the PR pipelines. We also use tools like ShiftLeft to check security problems as early as possible.
Exploratory testing is an unscripted technique used to discover unknown issues during and after the software development process. We should do exploratory testing throughout the testing life cycle. We generally start exploratory testing after the requirement analysis phase as early as possible. This way, we can find the risks and problems earlier, which also helps automation test efforts. We use many inputs and outputs of exploratory testing sessions for test automation. In API exploratory testing, we use postman and insomnia tools. For exploratory testing details, you can check this article.
Test Data Approach
One of the biggest challenges in testing is test data, and we are following below essential points which are also stated in our test data strategy:
- We have test data generation services and call those services’ endpoints to get freshly created test data for our scenarios.
- If we have no options to create the test data automatically, we do a requirement analysis and create it manually or get help from the relevant teams for our needs.
- We create new test data before execution and delete those after completion.
- In test environments, we mask critical and confidential data.
Pre-Release Testing Approach
Before releasing our services to our customers, we do some pre-production verifications. Before production deployment, the service team should verify to cover their new changes and critical scenarios.
If the service pipeline is green, tests for system integration, performance, user acceptance, and the final checks on the pre-production environment have passed, we are ready to proceed further to the production deployment. Whenever the deployment to the production environment has been done, and the new version of the service is being consumed in production, we monitor it with APM and logging tools.
For the previous article, please visit the below articles.
We have come to the end of the series of Microservices Testing Strategies. I have shared the microservices testing strategies that we are following at the Emirates Group. What are your strategies and opinions on microservices testing strategies? I would love to discuss this in the comments. If you are passionate about microservices and want to be part of something great, browse our current vacancies and find out what it is like to work for a dynamic and multicultural business.
#CodeEmirates #CodeBetter #DevRel
Lead Technical Consultant, Emirates Group