Tips for Software Performance Testing
General Tips
- Key performance metrics
When running a performance test, request throughput and latency are two key performance metrics that would help you to get an understanding of the performance of the system being tested. These are usually provided by the testing client. - Use an identical environment for testing
When it comes to Enterprise computing systems, it is a common practice to have multiple environments such as production, staging, development etc. It is recommended to have at least one environment identical to the production environment. One advantage of such an environment is that it could be used for running performance tests. Since it is not the production environment, testing won’t impact actual traffic. And since it is identical, exact conditions of the production environment are replicated during the test. - Collect system metrics during the test
System metrics such as load average, memory usage, network I/O etc… can help you to understand what is going on in the system when it is being tested. It could help you to understand why a system performed poorly during the test (Ex. System metrics may indicate that the CPU was a bottleneck). These metrics are also invaluable when carrying out performance tests for capacity planning purposes as they indicate which components need to be scaled. - Automate the tests
When evaluating the performance of a Software system, multiple tests are executed by varying different parameters. Ex. A system may be tested for different concurrency levels, message sizes, etc... Changing these parameters and running these tests manually becomes cumbersome after some time. Therefore take some time to automate the tests. You’ll thank yourself later. - Let the system achieve a steady state
Software systems usually take some time to achieve a steady state with a given load. Java programs are a good example for this where applications achieve a steady state after Just-in-time compilation. Therefore if the duration of the performance test is too short, the system might not achieve a steady state. The performance of a system cannot be understood properly before it reaches a steady state. - Remove the warmup period from results
This is related to #5. A system may not reach its potential performance level until it reaches the steady state. Therefore having the period before the system achieves the steady state, the warmup period, in performance test results may cause it to be erroneous. Therefore remove the warmup period from test results. Ex. Run a Java performance test for 30 minutes and remove the first 5 minutes as warmup.
If you are using Apache JMeter as the performance testing client, JTL splitter in [1] can be used to remove a given warmup period from JTL files. - Test components individually to identify bottlenecks
Unless it’s a monolithic system, a complete Software system would be made up of many interconnected components. A performance test of the complete system would look at the system as a black box and indicate the overall performance of the system. At times, such a complete system test may give you unsatisfactory performance results. If that is the case, components must be tested individually or a subset of components must be tested at a time to identify the bottleneck.
Consider a system with 2 microservices and a Database as shown in Figure 1.
To the test client, the system is a black box. It sends a request to an endpoint (Microservice 1 in this case) and receives a response. A perform test of this nature cannot pinpoint bottlenecks. If you need to identify bottlenecks, the performance of each component/subset of components in the system must be tested.
Firstly we can bypass the microservices and test the database directly as shown in Figure 2. We can take the request that we sent to microservice 1 and use that to derive the database query microservice 2 would send to the database. The test client can then be configured to test the database with this query.
This type of a test can be used to understand the performance of the database. Ex. Are select queries slow due to a large number of rows? Is the Database server able to handle a concurrency of this level with this type of a query?
Next, we can again use the request that we sent to microservice 1 to derive the request that microservice 1 would send to microservice 2. Then we can configure our test client to send that request to microservice 2 directly thereby testing a subset of components (microservice 2 and Database) of the system. This scenario is depicted in figure 3.
Suppose that this subset of components (Microservice 2 + Database) showed a significant performance drop compared to the direct database test. Also suppose that the overall system performance was similar to the Microservice 2 + Database performance test. Then it is likely that microservice 2 is the bottleneck.
However, one must always remember that not all bottlenecks need to be fixed. A bottleneck might be normal in a component due to the nature of work that it does. Even if the bottleneck in microservice 2 was fixable, the bottleneck might then shift to microservice 1. This too has to be verified by repeating the overall system performance test. Therefore, to pinpoint bottlenecks, subsets of system components have to be tested gradually.
JMeter specific tips
Apache JMeter is a popular performance testing client. Therefore I have included few tips for it as well.
- Design with the GUI, test with the CLI
JMeter has two interfaces. A GUI and a CLI. Use the GUI to create the test plan and to validate it. But once everything is working fine and you need to run the actual performance test, use the CLI. This is because the CLI mode uses less computing resources. - Remove unnecessary elements from the test plan
Elements such as “View results tree” in JMeter are good for checking if the test plan is working fine. But using them in your test plan can use more resources. - Use distributed testing to generate heavy loads
Normally, a single instance of JMeter is run and it sends requests to the system being tested. But if you need to generate large loads from JMeter (Ex. 1000 concurrent users) it is better to carry out distributed JMeter testing. With distributed testing, multiple JMeter nodes are configured to send requests to the system being tested.
