One year with framework independent Black Box tests via Testcontainers

As every developer knows, automation testing is an integral part of every software product and importance of right approach to their development is as important as the approaches to the development of the application itself.

In this article I would like to talk about the Black Box approach to testing an application, what are the advantages and disadvantages of this approach, and how to test your application with Testcontainers in this case.

Problems with the most of automative tests written by the developers

Usually developers use framework’s tools such as Spring Boot Tests for testing applications. It isn’t bad but in some cases has two basic disadvantages:

• Tests become framework-dependent which increases the complexity of migration to other frameworks
• Test fragility increases because usually developers use many source classes in testing code and when a structure of classes change, you need to change tests also

I often saw and wrote such tests, and everyone in the team understood that if we wanted to change framework or project structure, we would have to rewrite a ton of tests, which are even more than the source code itself.

A year ago we started creating a new product, which three months after the start of the development should have already been launched. We didn’t have enough time for developing the business logic and searching for optimal structure of our services because business requirements and our product changed and evolved very quickly. Of course, typical story for startup. We decided to use the simplest design for the services and to refactor it when the time comes. Also we wanted to write framework independent test to be able to migrate service to another frameworks in the future. But in order not to be afraid of the continuous refactoring process, all of our tests had to be stable and not affected by changes in internal source code.

As usual, we didn’t have enough time to develop unit tests for each class along with integration tests, so we decided to write for some classes unit-test and the most part of our application cover with integration tests using the Black Box approach, which was recommended to us by our colleagues from another team.

Black Box testing with Testcontaines as a basic approach to writing autotests in an application

What is Black Box testing? Black Box testing is an approach when you test your application without peering into its internal structures or workings.

How can we use Black Box testing in a software application? It’s simple. We just need to make virtual environment for our application and run it as you run it in a production environment. For example, if you have REST API application, you need just to call your API from tests and check result.

Of course, you can use Spring Boot Test and RestTemplate for testing your API but as I said earlier we wanted to write tests which could easily migrate to other frameworks. In this case we decided to create Docker container with our application and run it via Testcontainers together with other containers such as Oracle and Kafka.

First of all we need to create GenericContainer with our application:

 public static class ApplicationContainer extends GenericContainer<ApplicationContainer> {
        private static final int GRPC_PORT = 9091;
        private static final int HEALTH_CHECK_PORT = 8085;

        public ApplicationContainer() {
            super(image());
        }

        @Override
        protected void configure() {
            super.configure();
            withExposedPorts(HEALTH_CHECK_PORT, GRPC_PORT);
            waitingFor(Wait.forHttp("/system/readiness").forPort(HEALTH_CHECK_PORT));
            withStartupTimeout(Duration.ofMinutes(5));
        }

        private static Future<String> image() {
            Path dockerfilePath = Paths.get(System.getProperty("user.dir"), "Dockerfile");
            return new ImageFromDockerfile("test-app", true).withDockerfile(dockerfilePath);
        }

        public int getGrpcPort() {
            return this.getMappedPort(GRPC_PORT);
        }
    }

The above code shows creating application container from Dockerfile. For testing we use same Dockerfile which then use for deployment to production environment.

Then you can define your container along with the other and bind them with common Network:

private static final Network network = Network.newNetwork();

public static final OracleContainer oracle = new OracleContainer("gvenzl/oracle-xe:18.4.0-slim")
        .withNetwork(network)
        .withNetworkAliases("oracle")
        .withDatabaseName("ORCL")
        .withUsername("user")
        .withExposedPorts(1521)
        .withLogConsumer(new Slf4jLogConsumer(LoggerFactory.getLogger(OracleContainer.class)));

public static final ApplicationContainer applicationContainer = new ApplicationContainer()
        .withNetwork(network)
        .withNetworkAliases("app")
        .withEnv("DB_URL", "jdbc:oracle:thin:@oracle:1521/ORCL")
        .withEnv("DB_PASSWORD", oracle.getPassword())
        .withEnv("DB_USERNAME", ORACLE_USERNAME)
        .withLogConsumer(new Slf4jLogConsumer(LoggerFactory.getLogger(ApplicationContainer.class)));

Please note that there using network aliases for creating connection between services inside Docker (jdbc:oracle:thin:@oracle:1521/ORCL)

Of course, when you test your application from a container, consuming logs is very useful so we use Slf4jLogConsumer for this:

.withLogConsumer(new Slf4jLogConsumer(LoggerFactory.getLogger(ApplicationContainer.class)));

After that you can write your tests code via usual tools such as BlockingStub for gRPC or OkHtpp3 for REST API or KafkaConsumer and etc. As example for testing gRPC we create channel with parameters from applicationContainer and call requests like that:

@ExtendWith(Containers.class)
public class GrpcTest {
    private final TestServiceGrpc.TestServiceBlockingStub grpcService;

    public AgreementManagementTest() {
        ManagedChannel channel = ManagedChannelBuilder.forAddress(applicationContainer.getHost(), applicationContainer.getGrpcPort()).usePlaintext().build();
        grpcService = TestServiceGrpc.newBlockingStub(channel);
    }

    // JUnit tests here...
} 

To work with the database we use a pure HikariCP, create a connection, then run the query we need. In the case where we test kafkaProducer, we just need to create a new record in a certain table in tests, and then through kafkaConsumer expect the necessary records to appear in the topic through awaitility.

So now we try to design our services that they can be easily tested through this approach.

Results

During the year we continuously improved our services, some even changed frameworks. But throughout the year we did not rewrite our tests. On the contrary, they helped a lot not to be afraid of changes and to test them right away.

With the release of Spring Boot 3, we moved one of our services to the Native Image build. In terms of testing, we only needed to change the dockerfile and its assembly. Moreover by switching the dockerfile we were able to test the applications on both the JVM and the binary.

To summarize, I can highlight the main advantages of this approach to writing autotests:

• No dependence of tests on a framework made it easy for us to migrate to other technologies
• The refactoring process has become simpler and faster, since now we can completely change the internal structure of the project without thinking about rewriting the tests and without the risk of breaking anything
• The value of tests has increased because we create an environment similar to a real one and test the full cycle of an application

Of course, there is a number of disadvantages of this approach:

• The complexity of test development is greatly increased by the fact that we can not use Mockito and have to fill the database with all the necessary DDL and DML operations. In the most complex processes this is not feasible
• To run the tests, you must configure the environment to be as similar to the real thing as possible. Sometimes this is very difficult to achieve. Some parts of the system using proprietary external services are difficult to find in the docker image. They do not lend themselves to testing. For example in our case we use Microsoft SharePoint

A lot has changed within our product over the year, and this approach has successfully proven itself to the project and has stood the test of time.