Functional Tests for Docker Microservices

Testing services in isolation using docker-compose

Photo by: Jensen Stidham on www.shaw.af.mil

Managing a microservice code with unit tests and testing it in a live-like test environment with other components of the application is good practice, but sometimes it’s just not enough.

Unit tests are great; they help you to write better structured code that is easier to maintain and understand. Unfortunately it can happen that a service with a good unit test coverage doesn’t actually start when deployed.

Testing in a live-like environment is also a useful approach. In these tests the service interacts with other microservices, the monolith (of course), a real database, messaging queue, etc. — exactly as in production. As the complexity of an application increases, such environments get less stable, more difficult to maintain and eventually become a bottleneck for Continuous Delivery.

It’s usually quite difficult to test all the use cases in this manner, not to mention corner cases, error handling, etc. As a result, many components of the application don’t have a good set of regression tests, which makes it daunting to change and improve them.

Testing in a small & isolated environment

What if we create a small, fully-isolated test environment for our service that is completely under our control? This way we could simulate more use cases by interacting with all the involved components directly. For example, we could:

• directly send any requests to the service being tested
• save test values into the database or verify the values stored by the service
• publish test messages to a RabbitMQ queue or Kafka topic directly and check how the service handles them -or- consume and verify the message produced by the service
• replace some services with mocks and make them respond with custom data, respond with a delay, refuse connections or verify requests coming from the service under test

With Docker, it should be easy to create such environments. Luckily, most of the popular services like MySQL, RabbitMQ, Mockserver, and many others have ready-to-use images in Docker Hub and there is a fantastic docker-compose tool which we can use to start multiple containers at once.

Implementation

Sounds easy! Let’s write some code. For this example, I’m going to use Node.js, but it shouldn’t be a problem to do it in any other language. All we need is to:

1. create a docker-compose configuration file
2. write a script which will use a docker-compose command line tool to manipulate the containers and run tests

Simple case

In the most basic case, the service being tested doesn’t use any other services. Let’s create a simple service to play with:

Service code.

Service Dockerfile.

The docker-compose configuration file which we will use to build and run our container will look like this:

Here is the script which will start the container (our environment) using docker-compose cli, run tests using Mocha as a test runner, and then stop the environment:

The script for manipulating containers and run tests.

Simple test.

As you may have noticed, we used Mocha programmatically in order to be able to pass some data to the tests (ctx.serviceBaseUrl in this case). If you choose to use a test runner which can’t be used programmatically, such data can be passed via environment variables.

The overall file structure of the repository will looks like this:

service.js
Dockerfile
test
test/unit
test/functional
test/functional/tests
test/functional/tests/test.js
test/functional/docker-compose.test.yml
test/functional/environment.js
package.json

Let’s run the code and see if it works:

Looks good. Now let’s review what problems this solution has:

1. Starting the service container takes time. It may be a problem if we need to run the tests multiple times when we write them. One solution could be to start the environment once, and stop it only if we don’t need it anymore.
2. If we update the service code, we will need to rebuild the image and restart the container. This will slow down the development process if we want to use a TDD approach, for example. To improve this, we could map the service source code to the container as a Docker Volume and use the nodemon tool to restart the Node process inside the container automatically when the code is updated.
3. We build our own image of the service. It would be nice if it was possible to test the exact Docker image which will be deployed to production. The solution would be the possibility to specify a Docker image to use in docker-compose.test.yml.
4. We use static external port for the container. This may cause port conflicts if we decide to run our tests in the CI environment. The solution here would be to change the ports configuration in docker-compose.test.yml to use dynamic external port and use docker ps command to find out the port number which was assigned to the container.
5. There is no way to measure code coverage. As the service code runs inside the container we would need to instrument it in order to get code coverage information. To do this we can leverage the Nyc tool.

Let’s improve our framework before moving to more complex service examples.

Simple case improved

In order to fix the problems we discovered in the previous solution we need to create separate docker-compose configuration files for:

• CI mode, when the particular image is tested before releasing to production
• development mode, when the service inside the container is updated and restarted as we edit the source code
• measuring code coverage

Luckily, docker-compose accepts more than one configuration file at a time so we can avoid duplication:

docker-compose -f base-config.yml -f case-specifig-config.yml up -d

Base configuration file.

The configuration to create a test container from an image.

The configuration to be used in development mode.

The configuration for measuring code coverage.

The improved environment.js script will look like this:

Notes:

• The container dynamic external port is discovered by parsing docker ps command output. We can get the port using docker-compose port command as well, but it’s much slower for some reason.
• The code coverage report is created by sending the nyc process the SIGINT signal. As the source code directory is mapped to the /app folder inside the container, the report files are available in the testdata directory on the host.

Let’s try out the improved solution:

starting the environment

running tests

getting code coverage report

100% of the code covered, awesome!

stopping the environment

Works as expected!

Let’s now look at the cases where our microservice communicates with other services.

Service discovery

If you use a service discovery mechanism the service under test should be updated to search for the test instances of the services. This can be done using environment variables. The following is the example for Consul + Registrator case:

If you have Consul and Registrator running in your development and CI environment, the waitForServices part of environment.js can be updated to wait until Consul health check passes after starting the containers.

In case of DNS based discovery the target service domains can be overridden with Docker Compose service names which can be resolved to the container IP addresses within the Docker network.

Using the database

It shouldn’t be a problem to find a Docker image for the database since most of the popular solutions have been published to Docker Hub.

In most cases, we need to wait for the database to be initialized and the best approach is to keep trying to execute a query until it succeeds. If the database structure is created by the service under test or imported from files by the database container itself, you will need to run such a query against the entity which is created last.

If the database structure is created by the service, you probably want to start the database first. This can be done by passing the database service name to the docker-compose up command: docker-compose up -d mysql

The database state should be reset between tests, so that they don’t affect each other.

Here is a simple example of the setup for the service which uses MySQL database:

the service code

docker-compose configuration

environment.js

simple test

Let’s run the test:

As you can see, we start the database container first and wait for it to be initialized before starting the service container which creates the database structure.

Mocking HTTP services

HTTP services can be mocked out using MockServer. It provides a convenient way to mock responses from HTTP dependencies and verify the requests sent by the service under test.

Service mocks should be reset between tests, so that the tests don’t affect each other.

The following is a simple example of a test using MockServer JavaScript client:

Tips and Tricks

• If you reset the database or MockServer state in tests, make sure that they are executed sequentially. For instance, Jest runs tests in parallel by default.
• It’s safer to bring all the components into the required state before each test vs cleaning up after tests.
• You can override the default command in the docker-compose configuration to do something before the container process starts: sh -c 'echo token=$$TOKEN >> config.cfg && ./service -c config.cfg'
• Use docker-compose logs ${service} command to get container logs

How it works in Pipedrive

The approach described to run functional tests has been widely adopted by development teams in Pipedrive.

We have moved common functionality and helper methods of frameworks to a separate repository and then distribute it as a private NPM package. The repository has helper methods for Consul, Docker Compose, MySQL, Kafka, RabbitMQ, functionality for logging, re-trying, etc. It also includes many minimal working examples for different framework setups, like the ones presented in this article.

Functional tests are executed by Jenkins jobs while building Docker images or NPM packages.

Conclusion

As you may have noticed, the described approach for testing Docker microservices is very simple and flexible: The Mocha framework in the examples above can be replaced by Jest; WebdriverIO or Puppeteer can be used to run UI tests; the service under test and tests can be written in different languages; and you are not limited to functional tests only, the same way you can do integration, load, performance or stress testing. Keep experimenting!