The cost of replacing H2 with Testcontainers in a monolith with debt

The increase of tests duration after replacing H2 DB with Testcontainers on a real life example

Recently, we have taken over a small Java microservice. It had a well-defined domain, however the logic was quite complex and the tests covered only selected pieces of it. The fact of being responsible for further development and maintenance of the service imposed on us adding more automated integration tests. As a result of it we increased our self-confidence when working with the service. Moreover it was natural opportunity to understand the business logic and get to know the domain better.

Trial of Testcontainers

Usually our first-hand choice of an approach for testing data layer in Java applications is H2 in-memory database. It’s extremely fast and its configuration is trivial. However, in this handed-over component case, we encountered an issue with DB migrations. The syntax of some of them was not compatible with H2. In this circumstances, instead of adapting the migrations to H2 (which didn’t seem wise as our first job in a new domain) we decided to go with a database module from Testcontainers which guarantees complete compatibility with a real database. That was a chance of trying something new and today we are satisfied with the result. It’s working really well, the configuration wasn’t tough and as of the date of writing this article we are still using it.

Thought— use it everywhere

Using Testcontainers with a success in one place led us to start thinking about replacing H2 DB in tests of other components. The main advantage of this move (and most often mentioned in the context of Testcontainers) is to increase chances of discovering database related bugs and misconfigurations. H2 will always be an emulation while a container reflects behaviour of a real database. Consequently the risk of deploying invalid code decreases. Besides that, we considered replacing H2 for other reasons, based more on our experience:

• we were unable to run UPDATE queries with JOIN. Although the issue occured in tests context only, we were forced to adjust application’s migration script to match H2 syntaxt. In the end we used nested SELECTs which is much slower operation
• we noticed quite late the default fracational seconds precision of TIMESTAMP type differs between H2 and MySQL. For H2 the default is 6 and for MySQL it’s 0. This caused errors in our application’s tests when we tried to switch to MySQL module from Testcontainers. And even more importantly — we realised there’s a discrepancy in time precision, between tests’ context and production environment!
• null check validations do not work well on embeddable fields

As the next component to apply Testcontainers in, we selected the largest one and most developed service in an area we are responsible for. The service can be perceived as a monolith, since it handles two main functionalities of the product, which make it a central point in the ecosystem. For better understanding of what kind of an application I am writing about, let me list some metrics below. Some may look irrelevant, others not that important for this report, but as a whole they should make the reader more aware about the application.

Characteristics of the service

Technological stack:

• Java 11
• Spring Boot 2.4.9
• MySQL 5.6
• Kafka 2.6

Metrics:

• source code lines: 63k
• number of endpoints: 41
• number of DB tables: 48
• lines of code covered with tests: 91%
• number of tests: ~900, tests with Spring Boot context: ~500

if you’d like to know about another dimension, let me know in comments

Testcontainers configuration

Enabling Testcontainers in Spring Boot application is quick and simple. What we had to do was to:

• add dependencies

testImplementation "org.testcontainers:testcontainers:1.17.3"
testImplementation "org.testcontainers:mariadb:1.17.3"

• adjust tests configuration

spring:
  datasource:
    driver-class-name: org.testcontainers.jdbc.ContainerDatabaseDriver
    url: jdbc:tc:mariadb:10.5:///?TC_DAEMON=true
    # driver-class-name: org.h2.Driver
    # url: jdbc:h2:mem:db;MODE=MySQL;DB_CLOSE_ON_EXIT=FALSE
  jpa:
    properties:
      hibernate:
        dialect: org.hibernate.dialect.MariaDB103Dialect
        # dialect: org.hibernate.dialect.H2Dialect

• adjust pipeline configuration — in our case we followed suggestions for GitLab CI available here

Result

After adopting Testcontainers in the other service before, we were sure it would be working and the process should be smooth. And it worked from the very beginning, even the first test run was successfull. Hovewer, we were concerned about how long the test would have lasted after this change. And our worries have been confirmed, although we did not think the time would increase so significantly.

Average time of tests with H2 setup:

• locally: 7 min 27 sec
• pipeline: 12 min 03 sec

An average time of tests with Testcontainers setup

• locally: 10 min 53 sec (increase by 46%)
• pipeline: 14 min 55 sec (increase by 24%)

Summary

When we planned this, we knew without a doubt that replacing H2 with Testcontainers will slow down the test run. But we were thinking about seconds (tens of seconds) rather than minutes. We believe the main problem lies in the way our tests are implemented. We have too many Spring contexts bootstrapped during a single test run. There are more than 40 DB migration processes run within it. This is a derivative of too much responsibility of the service, tests’ architectural mistakes like not well separated base classes, too many MockBean and unnecessary DirtiesContext annotations used. This is a state we reached after developing the service for 2 years, by 2 teams and during the time a lot of requirements have changed, the service responsibility grew significantly and although we wouldn’t say the tests quality is low and a technological debt is out of control, it was not easy to keep cleanliness in tests. And today we have to face the consequences of our decisions. Life.

Although we implemented what we planned initially, we could not afford to extend the already long duration of the tests. We humbly decided that this was not the right time for such improvement. It has to wait until we pay some technical debt in tests off. Anyhow, we don’t have a sense of wasted time, since the verification of the idea was a short process which reminded us about more serious matter which we should focus on.

PS: And Testcontainers are successfully adopted in other areas including new initiatives.

I hope our lessons learned about adopting Testcontainers will help you with making a decision whether to use it at all, and if so, then when to do it.