Optimise Testcontainers For Better Tests Performance

Testing an application during development should be something that is not disturbing our work but supporting it. Having a clear distinction between fast unit tests, and complex, but slower integration tests is a good start for achieving a quick feedback loop. The faster we will get a result of the most relevant test, the quicker we can implement new features without breaking the application logic. Unit testing is great during core logic development, but from time to time, we need to take a broader view and use integration tests, to check, how our module is interacting with other ones. In this article, we will focus on how to efficiently use Testcontainers for integration testing. After reading it, you will know, how to reuse created containers, and how to speed up test execution.

Notes before start

In the samples below, all tests are written in Groovy with Spock framework. Dependency management is done by Apache Maven. It is needed to have Docker installed on the workspace. Also, basic knowledge about Testcontainers is required.

Use real module instances instead of emulated ones

A lot of projects that I was working with were using H2 or Embedded MongoDB as a database during integration testing. Their benefits are ease of set up while not differing a lot from production databases in usage, as usually, we are using abstractions like JPA. Reality is that every database has its own specifics and mechanisms, which makes a huge difference. Used in-memory H2 database behaved very differently than production PostgreSQL, causing that tests were not reflecting real cases. Using Testcontainers instead of in-memory implementations eliminates those differences. In this article we will use two cases:

• testing with a database instance, as Testcontainers are providing support for database driver that simplifies it
• using Testcontainers modules and Docker images while manually managing container lifecycle

Testcontainers performance

The most time-consuming operation for Testcontainers is its startup. Every image that is being used, needs to be fetched from Docker Hub. Fortunately, this is a one-time operation. After that Testcontainers requests Docker to create a container from an image, which requires allocating file system, creating a network interface, and finally executing all scripts that are starting container internals. On top of that, we might need to apply our customization, like database schema creation. For a long time, this setup was always invoked by Testcontainers on startup, extending the test feedback loop.

Testcontainers are using Docker CLI for creating container. Image Source.

Starting from version 1.12.3 we got a Reusable Containers feature, that significantly reduces container startup time. Basically, when container configuration remained unchanged, it means that the container instance might be reused. Be aware of that in case of reusability, containers are not stopped after tests are finished or JVM is shut down — it will remain working in the background, waiting for next time it will be needed.

Enabling Reusable Containers

To use Reusable Containers, make sure you are using the proper version of the library. Add dependency to your pom.xml with at least 1.12.3 version. Samples in the article will use PostgreSQL and Apache Kafka in the currently newest version, which is 1.12.5:

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>postgresql</artifactId>
    <version>1.12.5</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>kafka</artifactId>
    <version>1.12.5</version>
    <scope>test</scope>
</dependency>

The second step is to configure Testcontainers with testcontainers.reuse.enabled property. Properties can be set up within .testcontainers.properties file located in home directory (refer to the documentation to see, where exactly create this file). Keep in mind that this is an environment property, so it cannot be set up in .testcontainers.properties file on project classpath. Assign true value, so that feature could be used:

testcontainers.reuse.enable=true

These two steps, adding dependencies and configuring property, are common and required for all Testcontainers. The rest of the steps depends on the type of container and the way it is configured.

Testing databases

To visualize, how Reusable Containers are working, we need some code. Our module will be responsible only for one thing — providing a repository of customers. It will allow for saving and retrieving them by id. Proper working of this repository will be ensured with a test.

Test for saving and retrieving customer data. It will be used in our samples to visualise Reusable Containers.

We want to use PostgreSQL as our database. There are multiple ways to start it with Testcontainers, and we will look into some of them, starting from using DriverManager class.

Reusing container with DriverManager

Testcontainers are providing their own implementation of Driver interface, that requires a little change to the connection URL, but allows to set up a database automatically. It requires adding tc: in connection URL, just after jdbc:. If you need deeper insight, please take a look into Testcontainers documentation, in other cases, just keep in mind that in tests with DriverManager an overlay on PostgreSQL’s driver is used. Let's configure the connection in the tests setup method.

Setting up database connection using TC library. Additionally, we are using a trait, that is logging connection acquire time.

Used JDBC URL contains TC_REUSABLE property, that indicates the container created using this configuration should be marked as reusable. Together with environmental Testcontainers configuration made earlier, the container will not be stopped by Ryuk after tests are finished. After starting the test, application logs will contain information about starting a new container, and time required for acquiring connection (together with starting newly created container).

[main] DEBUG 🐳 [postgres:9.6.17] - Starting container: postgres:9.6.17
[main] DEBUG 🐳 [postgres:9.6.17] - Trying to start container: postgres:9.6.17
[main] DEBUG 🐳 [postgres:9.6.17] - Trying to start container: postgres:9.6.17 (attempt 1/1)
[main] DEBUG 🐳 [postgres:9.6.17] - Starting container: postgres:9.6.17
[main] INFO 🐳 [postgres:9.6.17] - Creating container for image: postgres:9.6.17
Execution time: 5706 ms

It is clearly visible, that a new instance of the container is being created, however, it is not stopped after tests are done. Starting test once more will produce another log with information about the reused container.

[main] INFO 🐳 [postgres:9.6.17] - Reusing container with ID: 920731ce978eb2867c8713ae9d6a4066d1fc15a3e2b2d60afe56da2a74286fa5 and hash: 051496e6089879e1bf505f0fba073d2610e4f232
[main] INFO 🐳 [postgres:9.6.17] - Container postgres:9.6.17 is starting: 920731ce978eb2867c8713ae9d6a4066d1fc15a3e2b2d60afe56da2a74286fa5
Execution time: 1632 ms

In this sample, the reused container appeared to reduce startup time over three times!

Reusing container with Testcontainers modules

There are multiple modules available with a predefined configuration of Testcontainers. All of them are able to be started with reusability flag, as GenericContainer class exposes a withReuse(Boolean) the method, that allows specifying, if this particular instance of the container should be reused or not. By default, the container will not be reused. It is important to define reusability prior to starting a container, as in other cases it will not take an effect.

Using the same test, but different setup of container. A little more manual work is needed.

Running this test will behave similarly to the previous approach: Testcontainers will check, if there is a matching container, and when a matching container is found, it will be used in tests, in other cases, it will create a new instance of the required resource.

Using the wrapper approach for Reusable Containers

While using Testcontainers with multiple test classes, you can use JUnit @Rule, Spock @Shared, or wrapper approach described in this article. Let’s use it together with the reusability feature for Apache Kafka. The starting point for this will be class made on top of KafkaContainer with a new container setup, that will be configured with reuse feature.

Starting Apache Kafka container with reusability feature enabled

Unfortunately, this is not enough. When tests are invoked once more, it is not reusing existing instances of containers, but always creates a new one. To understand, why it behaves like this, we need to look into the internals.

Determine if the container can be reused

All containers are created by commands. Finding a container, that can be reused is based on idempotence — a command that produces a container should always be the same for the same container configuration. It means that once a configured container should always produce the same command to start it. This command is converted into JSON object, and its hash is calculated and stored within Docker Container:

A part of command in JSON format used for creating a container.

Body of command contains some properties that seem to be random values, like aliases or hostConfig.NetworkMode. These generated values are changing the hash value calculated from it. With different hashes, Testcontainers are thinking that there is no matching container that could be reused, so a new one is created. To overcome this, it is needed to nullify networking configuration, as in default configuration with only one broker and internal Zookeeper it is not needed.

Setting network to null will prevent generation of random values, while container remains usable.

Distinguish containers of the same type

There is one more interesting thing in command that creates a container — labels. Why they are so interesting? Imagine a case, where multiple projects are using Apache Kafka, and are tested with Testcontainers. If all applications use the same container config, like image version, nullified networking, and reusability, then all of them will use the same container during tests. This might lead to problems with data quality, as it will contain data from multiple applications that might be conflicting. It would be helpful to have the possibility of running one container per application. Using custom labels, we can differentiate them.

Container can handle multiple custom labels

Multiple label names can be used with different values across all projects. Every label that is added to the configuration, will be reflected in command JSON, making it more unique. This allows for more extensive use, like container versioning, e.g. in case of intense development, when Apache Kafka topics are changing quickly, or database migration scripts are changing dynamically, you can include information about them in labels. It will allow for the creation of new containers each time data set is changed, and reuse existing containers when its data scheme remains.

Invoke custom actions only on new container

Usually, while starting a container, project-specific initialization is needed, like creating a schema, adding test data, or maybe invoking some endpoints. During the startup of the test environment, reducing the wait time to a minimum is desirable, but executing initialization operations every time connection to the container is acquired is taking a lot of precious time. The solution for this has come with version 1.12.4 when an indicator of container reuse has been added.

Lifetime hooks

During Testcontainer lifecycle multiple hooks are invoked. Hook methods, defined in GenericContainer class, are invoked in various states of a container in the following order:

• containerIsCreated(String)
• containerIsStarting(InspectContainerResponse, boolean)
• containerIsStarted(InspectContainerResponse, boolean)
• containerIsStopping(InspectContainerResponse)
• containerIsStopped(InspectContainerResponse)

Boolean flag parameter of containerIsStarting and containerIsStarted are holding information, whether the container was reused or not. It can be used to invoke initialization operations after the container has been started and is ready to use.

Using hook to determine, if container was reused or not.

containerIsStarted hook allows determining, if a container is freshly created, or was already created before. Invoking heavy, long term operations only on a new container can save a lot of startup time — the more initialization operations are invoked, the bigger the payback is. This feature can be put together with labels — every time label value is changed (like information about supported Kafka topics), a new container will be started and containerIsStarted hook will provide information about it, so customization can be performed.

Conclusions

Reusing containers can speed up test execution, but it requires to remember about some drawbacks. Keep in mind them before you will start to use it:

• Containers are not cleaned up between test suites runs, so it is required to randomize some test data (e.g. IDs stored in the database). It will prevent data conflicts.
• Labels can help distinguish containers between projects.
• Using database migration scripts might be problematic, as libraries are often validating their checksum. During development, those scripts might change, causing checksum mismatch.
• Reusability must be provided as environment property. It cannot be provided within the project configuration.
• Containers need to be stopped manually (or on host system shutdown). When not cleaned, it might lead to high memory, CPU or hard drive utilization. Keep your hands on, as this is going to be improved in future releases!

All presented code can be found on my GitHub.