Time and Memory Performance Verification Using JUnit
Abstract
Automating tests for functional requirements is a topic that has a vast set of documentation and books. But that is not the case for those of non-functional nature, even if it relates to performance, which may be a source of headaches on production stage. This article describes how we automated some performance tests in the hope of helping others or at least contribute to the discussion for better ways of doing it.
The Problem
In a current project we had to generate a report that exhibits annotated photos of industrial tubing settlements: that report was named the Image Report. Before this report can be generated, inspectors should walk through an industrial plant taking pictures of settlements of interest, many of them comprised of engines and connecting tubes of possible leaking material. Figure 1 is an example of a photo that illustrates such an area of interest that will be later annotated.
Figure 1 — Raw photo
In the back office functionality, after uploading that picture into the system, the inspector switches to an image editing mode where she can visually deploy pins attached to notes on identified spot points of potential leaking. Figure 2 show an example of an annotated picture.
Figure 2 — Annotated photo
Since these indications are dynamic, we opted for storing the original photo in a database blob field and the annotations in a common table. Before view time, the system assembles the final picture by combining both in a resulting picture, as per Figure 2.
After all those information are registered in the back office system, campaigns are made as to measure and apply correction to potential leaking spots as needed. This is where the Image Report takes place: it is generated to guide the work of inspectors on the field based on campaigns parameters. The Image Report is assembled with a number of images, each containing all or a subset of its annotations, and with an additional grid below the image providing additional information of the drawn interest points, as in Figure 3.
Figure 3 — Complete Report with Images and Annotations
The following non-functional requirements were specified for the report generation:
- Maximum of 200 images with an average of 50;
- Each image will have a maximum of 10Mb size;
- Each image will have a maximum of 20 annotations;
- There will be a maximum of 3 report generations spread over a day;
- The maximum time to generate will be up to 10 minutes for the worst case scenario and 5 minutes for the average scenario.
Since we host our app in a shared server, we also imposed a memory consumption constraint so that we would not lead the server to a shutdown due to a critical lack of memory. To couple with that, we established the limit of 800 Mb per report execution, which is reasonable for the server in question considering the maximum of 3 generation spread by day.
Test Implementation
Separate Performance Execution Suite
Given the peculiarity of this test, that would consume a lot of time and resources, we couldn’t allow it to be mixed up with the app’s unit and integration tests. It also doesn’t make sense to mix them as they have different objectives. So our first decision was to create a separate execution profile dedicated solely to this performance test. In our continuous integration server (Jenkins), we created a separate performance validation job that would not be executed in the integration build, instead it should be manually started. In our Maven configuration we added the following lines:
report-test
maven-surefire-plugin
2.13
**/*Performance.*
(...)
The performance validation job is set to execute the build process of the app with this profile up, so only tests with Performance suffix in its name get executed.
Preparing Test Scenarios
Before start measuring, we needed to define the test scenarios and prepare them with the necessary data to exercise the report generation. We defined four scenarios, where in each an Image Report is generated with:
- A total of 10 images, each with 20 annotations points, within at most 1 minute;
- A total of 50 images, each with 20 annotations points, within at most 5 minutes;
- A total of 100 images, each with 20 annotations points, within at most 8 minutes;
- A total of 200 images, each with 20 annotations points, within at most 10 minutes.
That way we believe we could simulate the most interesting scenarios, including the worst and average cases described in the non-functional requirements. The test code look like this:
Map cases = new HashMap() {{
put(10L, ONE_MINUTE_TOP );
put(50L, FIVE_MINUTES_TOP );
put(100L, EIGHT_MINUTES_TOP );
put(200L, TEN_MINUTES_TOP);
}};
for(Map.Entry scenario: cases.entrySet()){
executeTest(scenario.getKey(), scenario.getValue());
}
The actual persistence layer is executed using Hibernate and consumes test data from a test database. To simplify the mass data generation and avoid end up with an enormous set of database scripts, we used the business application methods to create the mock data on the test database — it would slow down the test execution, but greatly speed up the development. An unique 10 Mb image (worst case as specified in the requirements) is used in all records, since it would not matter to our processing time if they were the same. We then isolated those pre-tests preparation from the actual report generation, and then executeTest method looked like this:
executeTest(Long scenarioImagesQty, Long scenarioTimeLimit){
createDataSetWith(scenarioImagesQty);
executeReportGeneration()
}
We now needed to pu validations in place.
Time and Memory Consumption Validation
To make the time consumption validation we collect, inside the previously described executeTest routine, a common System.currentTimeMillis() just before and after the actual report generation, so we can compute the elapsed time to compare it with our predefined limit:
executeTest(Long scenarioQtdImages, Long scenarioTimeLimit){
createDataSetWith(scenarioQtdImages);
long t0 = System.currentTimeMillis();
executeReportGeneration()
long t1 = System.currentTimeMillis();
long elapsedTime = (t1 - t0)/1000;
String timeOutMessage = "Time limit exceeded for " +
scenarioQtdImages +
" images. Expected " +
scenarioTimeLimit +
" seconds, total time: " +
elapsedTime;
assertTrue(timeOutMessage, elapsedTime < scenarioTimeLimit);
}
A simple internet search shows that the use of System.currentTimeMillis() is very common for this situation, so we just added the assertion in order to automate the procedure and give feedback. On the other hand, analyzing memory consumption in the JVM is tricky because GC policies interfere in the expected results. For example, memory consumption could be measured just like time consumption:
System.gc()
memoryBefore = runtime.totalMemory() - runtime.freeMemory()
doStuff()
memoryAfter = runtime.totalMemory() - runtime.freeMemory()
memoryConsumed = memoryAfter - memoryBefore
The problem is that the GC may execute before the second call to totalMemory(), and then, even if the functionality has consumed 3Gb through its execution, the measure will show a low consumption, deceiving the automated test. We could also do something like described in [1] or mess with GC policies. Instead, we decided for a simpler way to determine if the report execution would be extrapolating the limit of 800Mb per execution: simulate an environment with default GC configuration but with memory limitation, specifically 800Mb, and see if the test execution blows it up or not. This is simple enough to do with Maven, we just have to configure surefire plugin inside the report-test profile with a -Xmx800m:
report-test
maven-surefire-plugin
2.13
-Xms200m -Xmx800m
**/*Performance.*
(...)
This strategy was enough for us and we believe to be enough for most general cases, but something else will need to be done if a more grained measurement is needed.
Conclusion
The automated test implementation described is a best effort towards an automated verificatin of non-functional requirements related to performance and memory consumption constraints. Things can change a lot in a production environment and we only wanted to measure the approximate performance behavior and find out, yet in development stage, the most non environment specific bottlenecks that would prevent us from complying to the non-functional requirements. It actually helped us because we could see that we had problems with both memory and time execution. With that information we could fire a Visual VM profiler against the performance tests itself, visualize the bottlenecks and think about solutions.
In the end we could adhere to the non-functional requirements, the clients were happy with the result and we ended up with a nice suit of automated tests that could be used anytime we wanted to assess the performance of the report either by looking at its result or by profiling its execution.
[1] http://stackoverflow.com/questions/19785290/java-unit-testing-how-to-measure-memory-footprint-for-method-call
Coauthored with Márcio Belo — marcio.belo.nom.br
