Wasn’t this faster before? Performance assurance by unit test measurement
Nearly every end user made this experience: During numerous version, a software is fast and works as expected, and then, suddenly or stepwise, the performance deteriorates. This is not only uncomfortable for the end user, but also causes higher consumption of computing resources, e.g. CPU time or network bandwidth, resulting in higher power consumption and possibly in end users abandoning the use of the tool, the app or even the product. One reason for performance regressions in released code is the absence of tests for the performance: Only ~3,4% of all projects contain benchmarks or load tests. Therefore, a lot of projects do not test their performance at all — and in order to assess the performance of a software, we need some workload for measurment.
What to Measure?
To measure performance without existing performance tests, we make the following unit test assumption: The performance of relevant use cases of a program correlates with the performance of at least a part of its unit tests, if the performance is not driven mainly by external factors.
While these may not create reliable results under certain circumstances (functional utilities interfering with measurement, unsuitable workloads), this makes it possible to avoid additional effort for creating performance benchmarks and it makes use of tests which usually cover large portions of the software.
But Doesn’t This Take too Long?
Only 62% of all projects get performance measurement results in less than 4 hours — hence, even if a benchmark exists, feedback creation is time-consuming. To speed up feedback, PeASS uses a regression test selection: For every test, it is determined whether the changes in the current version may influence the outcome of the test. This is done using a combination of static code analysis and analysis of execution traces. PeASS is able to reduce the number of executed tests to approximately 5% of all tests.
Will PeASS really find my performance regressions?
We examined the performance changes of 11 open source projects that are mentioned by the developers in commit comments. We found that 81% of all performance changes which are covered by unit tests can be found by PeASS. This only holds if certain characteristics are the same as for the examined open source projects — e.g. the performance of your software is at least partially driven by single threaded usage and if the test coverage is high.
Demonstration: Finding Changes in Apache Commons Fileupload
As an example, we used PeASS to analyze Apache Commons Fileupload, a widespread library for managing file uploads in Java. We found a performance change which slows down processing of uploads.
Maybe your software also has performance regressions in its version history you do not know — so give PeASS (https://github.com/DaGeRe/peass) a chance.
This blog post described the ASE 2019 Tool Paper “PeASS: A Tool for Identifying Performance Changes at Code Level”.
