Simplifying Integration Testing with Testcontainers and CircleCI

Assuring our code works as intended is crucial to us at mitiga.io. Until recently, our testing consisted of thorough unit tests, with a staging environment for manual integration tests. While unit tests are effective, they don’t provide as much value when interacting with other components, such as databases and other services. On the other hand, maintaining a live integration environment is becoming increasingly complex; eventually, we will end up testing the integration environment, rather than what runs in production.

We needed a way to isolate integration tests from one another, while remaining agnostic to the execution environment, developer machine, or CI. Using containers, which are stateless and isolated, seemed to be the ideal solution for these requirements.

At mitiga.io, we write our microservices in TypeScript. We wanted to create a thin wrapper around containers to ensure they are spun up before we run the tests and shut down when finished. Using the testcontainers framework seemed to be perfect for our needs. It manages containers and only hands control to the test code when the container is ready. It also has a monitor container that shuts down any leftover containers if your code doesn’t gracefully shut down.

We use CircleCI as our provider, because of its great integration with Docker and ability to enable layer caching on a per-project (repo) basis. In order to enable it, we need to add the setup_remote_docker step to the configuration:

However, there was a minor issue: there is no network connectivity between the CI container that runs the build, tests, and the integration container that runs Postgres, for example. This blocking of network access is intentional and for good reasons, but it makes our work more difficult.

We had a few options:

1. Make our tests run in their own containers, along with the integration containers in the remote docker environment.
2. Abandon using CircleCI’s remote docker feature and run DIND (docker in docker) in our build container.

Both of these options had significant drawbacks. Making our tests run in their own containers would require us to completely redo our CI configuration. Running DIND would result in longer testing times, because we would lose the layer caching benefits of CircleCI’s remote docker support.

We wanted more options. Our assets were: the build and tests running in the CircleCI container and a remote docker host with only the docker daemon port and SSH accessible from the CircleCI container.

Having SSH connectivity to the docker host provided a solution: we could open an SSH tunnel for every exposed container port.

Our solution required us to wrap testcontainers with our own code, so that we could control both the lifecycle of the containers and the SSH tunnels. For example, we had to open the tunnels before starting the containers, because the testcontainers library only hands control over to the test code after it verifies that the exposed ports are accessible and responding to connections.

Other challenges included:

• Handling testcontainers monitor container (called Ryuk) that removed leftover containers
• Having connections made to the localhost tunnel instead of the remote docker host
• Shutting down all tunnels when no more containers are running

Putting it all together

Below is a simplified working version of the steps needed to run tests, with an actual Postgres instance running in remote docker on CircleCI.

This is a TypeScript example using Jest as testing framework.

1. Update CircleCI configuration

Add setup_remote_docker section to enable remote docker (lines 12–14).

This configuration will both enable remote docker host and add ~/.ssh/config file with SSH access information to that host.

2. Install NPM dependencies for the example

3. The test we want to conduct: a simple Postgres query

This test spins up a Postgres container, returning the localhost port on which connections can be made (line 3), then connects, conducts a simple query, and tests the output.

4. startPostgresContainer() and friends

First, we determine whether we are running on a local machine or CircleCI (line 36). Running on the local machine is straightforward, according to the testcontainers documentation (lines 1–5, 7–10). Let’s examine how it works in CircleCI.

We find two unused network ports (line 26)

• One is used to connect to Postgres
• The other will be used by testcontainers to connect to Ryuk, which is a container that monitors created docker resources. When testcontainers exit or crash, Ryuk cleans up the resources (see here).

We create tunnels from the container running the test to the remote docker host on the ports we found in the previous step (lines 28, 21–24)

• We need to keep track of the open tunnels and save them in the openedSSHTunnels (lines 22, 23) array, so we can gracefully close them later.

Now we have to tell testcontainers not to assume the configuration from the docker client, as it will try to connect to the firewalled remote docker directly. We do this by setting environment variables to override the defaults (lines 14, 18).

Finally, we can spin up the Postgres container, telling testcontainers to map our local port (via the tunnel) to the container port (line 31).

5. Description of the utility functions used by runPostgresInCircleCI()

isRunningInCircleCI(): Are we running a local machine or CircleCI (lines 1–3) is done by evaluating the environment variable set by CircleCI

getRemoteDockerSSHConfig(): Parse and process the SSH config generated by CircleCI when remote_docker is enabled (lines 5–8). This information is later used when creating the tunnel to the host:

• The IP address of the remote docker host (line 15)
• The port on which the SSH daemon listens on the above IP (line 16)
• The username to connect to (line 18)
• The private key used to make the connection to the remote SSH (line 17)

createSSHTunnelToRemoteDockerHost(): Uses the configuration information supplied by CircleCI and the local unused port we previously allocated to create an SSH tunnel from the container running the test to the container running on the remote docker host.

• Notes: we are using the same port on the CircleCI container and on the remote docker host for our test — this is due to testing testcontainers limitations in the available configuration.
• line 20 refers to the remote docker host external port that is mapped to the running container
• line 22 refers to the port on the CircleCI container running the test port on which the tunnel listens

6. Full code sample