Optimizing integration tests written with JUnit 5
One day I had enough, after merging my branch into master I had to wait around ten minutes before I could deploy it. Ten minutes might not seem that much, but when you do several deploys each day, ten minutes will not be just ten minutes any more - it will rather grow and might be up to an hour on a good deploy day.
In PriceRunner’s backend landscape we develop and manage hundreds of microservices, which means several smaller updates and iterations on each application in order to have them up to date. Each application consists of unit tests as well as integration tests. These should then be executed locally and on our build server. Waiting for these tests to be executed may be time consuming, time that could be spent elsewhere.
It got me digging around to see if it was something in particular that took time, quite soon I realized that most of the time was indeed spent on the test stage. In this post I’ll describe how we, with a couple of naive and small efforts, did decrease the overall time spent on the test stage with about 43 %.
Profiling the problem
Our test stages are triggered when building the application with maven and the profiling was simply made by looking at the output from maven.
Environment Test Type Total time
--------------- ------------------- ---------------
| Local | Unit tests | ~0:30 minutes |
| Local | Integration tests | ~5:30 minutes |
| Build server | Unit tests | ~0:33 minutes |
| Build server | Integration tests | ~5:45 minutes |After the profiling was done I had some numbers to work against. First thing noticed was that we did not have any major differences when the tests were running on my local machine as on the build server, but we could see that most of the time were allocated to the integration tests. It indeed look like something is sketchy here, over five minutes on integration tests seem a bit much.
Looking back just a few weeks on our build history, I could see that the overall build time had increased with a few minutes, the overall build time was actually below 5 minutes back then. After we’ve added a few features which required new integration tests, the time did increase by a noticeable amount.
Conclusion: Time spent on our integration tests does increase more than it should. Fixing this will decrease the overall time by a fair amount.
Integration test setup
In most of our application we’re using JUnit 5 as our testing framework. One part of the JUnit 5 release was declarative extension which is great when writing integration tests, it easily lets you reuse common code to start one of your services that you need to integrate with, for instance a database.
@ExtendWith(DatabaseExtension.class)
class MyIntegrationTest {
// …
}In the above example, the test was only extended with one extension — in real life we commonly start multiple services to fully test our application.
In order to start services with the actual version and implementation that we use in production the services are started within docker containers. Behind the hood that is done by using the Testcontainers - a great framework for spinning up docker containers in your java application.
The test extension
The extension can implement several common interfaces, such as BeforeAllCallback, AfterAllCallback, BeforeEachCallback and AfterEachCallback. For those familiar with JUnit most of these are probably straight forward. The callbacks are invoked before or after either all or each test, `BeforeEachCallback` is for instance equivalent to the `@BeforeEach` annotation and will be invoked before each test. These can be used to start and stop services needed for each test and for instance to clean up the database between each test.
However, starting and stopping docker containers (and other services) for each test class might be a heavy process and might possibly be one reason why the integration test lifecycle takes so long time. An easy way to see if that might be the case is by looking at the logs when starting one of our docker containers.
[INFO] [10:09:59] — Starting an elasticsearch container using [docker.elastic.co/elasticsearch/elasticsearch:7.5.1]
[INFO] [10:10:16] — Container docker.elastic.co/elasticsearch/elasticsearch:7.5.1 started in PT16.822013SAround 17 seconds to just start one of our containers for one test, there is no doubt there is room for improvement here.
Optimize the extension
As I mentioned earlier, we do start and stop all services between all test classes which is the big problem here. If it takes, let’s say, 30 seconds to fire up our test extension, each test class full lifecycle will then take at least that time to execute. So by adding a new integration test class, another 30 seconds will be added to the overall time.
Test suites to the rescue? Test suites is great for this case but unfortunately JUnit 5 does not support test suites yet. You can get it working by using the JUnit 4 as explained by Sam Brannen here, but then you have to mix JUnit 4 with JUnit 5.
Hopefully JUnit 5 will support declarative test suites soon, but until then we decided to test an alternative approach. The idea is to only start our services the first time we enter the BeforeAll method in our first test and not close them down until we are done with our last test.
This is done by having a static instance handling if we have started our services our not and then add our services to the root context store. The root context will close down after all our defined tests and when it closes down it will invoke everything in its store that implements the ExtensionContext.Store.CloseableResource interface.
The TestContext class starts all services we need as well as implementing the ExtensionContext.Store.CloseableResource interface so when the root context shuts down its close method will be invoked and all services will be properly closed. TestExtensionInitializer is thread safe and responsible to handle if we should initialize the context or not - so it’s only done once. If we need access to the TestContext class (we might need an IP to a service etc.) in any of our tests the ParameterResolver interface can be added to the extension and the resource can then be retrieved from the root store.
Results
Only by this minor trick, we managed to get rid of most of the time spent on starting and closing services. As you can see below, we actually decreased the time spent on the whole integrations test lifecycle by 50 % (locally). But most important, we will no longer add the startup/shutdown time when adding new test classes, so the time should not increase that drastically any more.
Environment Test Type Total time
--------------- ------------------- ---------------
| Local | Unit tests | ~0:30 minutes |
| Local | Integration tests | ~2:15 minutes |
| Build server | Unit tests | ~0:33 minutes |
| Build server | Integration tests | ~2:24 minutes |Next step would be profiling the individual tests and check if something takes more time than it should. Here I know we have at least a few tests that is time consuming and where we could cut a few seconds - but that will have to wait until next time.
Markus Dybeck
