How Docker can help us with cloud services integration testing

What is software testing?

In general, under software testing, we consider a process of executing a program or application with the intent of:

• first, finding failures. Worth to mention that testing can’t identify all the failures within software, in this sense it would be more correct to say it establishes that software functions properly under specific conditions
• second, finding faults. It is important to distinguish between faults and failures, because not all faults will necessarily result in failures (e.g., faults in unreachable or dead code will never result in failures). Also, worth to mention that some faults may result in failure under very specific conditions, like environment is changed (e.g. typical phrase from software developer you can hear — this was working on my laptop), hardware platform is changed, or 3rd party software with which we are interacting is changed
• third, ensure it meets the requirements that guided its design and development, and acceptance criteria’s
• fourth, performs its functions within an acceptable time. Under this we consider software performance under specific conditions
• also, there are can be a lot of other things, like: usability, portability to intended environments, scalability and etc.

The Importance of test automation.

From what we discussed above we can see that testing can be a complex process. If you want to keep pace, you’ll have to look into ways to deliver your software faster without sacrificing its quality. CI/CD is a way you can solve this problem. Continuous delivery allows you to automatically run tests in build pipeline and deploy software to testing and production environments.
But which kind of testing do we need (and can we execute) in pipeline?

Perfect Test Pyramid

Lets look into image bellow. The test pyramid reflects different layers of the testing process, it’s complexity and time consumption. Mike Cohn came up with this concept in his book “Succeeding with Agile”.

We will not go deep into details of each layer, because it is out of the scope of this article, we will focus on aspects that are related to our topic.

The foundation of your software test suite will be made up of unit tests. Unit tests ensure that a certain unit of your codebase works as intended. Usually, the amount of unit tests dominates over other test types. It is worth mention that unit tests usually will be represented by a single function and can be well isolated from other components your system. So, you don’t need to struggle with external dependencies.

More complex part is the next layer which represents different kinds of integration tests, which involve a lot of external dependencies. Such dependencies can be databases, external APIs, cloud buckets etc. In terms of integration testing, you would like to test such interaction as well and it becomes nontrivial task. Especially it becomes more challenging when your software is meant to work in a cloud environment with cloud provider-managed services.

Cloud services testing

Let’s consider a simple system. We have 2 AWS lambda functions written on .Net Core and ElasticSearch service.

Here and further, we will use both names ElasticSearch service and OpendDistro for the same service.

The first function is responsible for the ETL process to provide data for the ElasticSearch service. The second one is responsible for processing some client requests to search for something within the data in the ElasticSearch.

Which challenges can we see from this diagram?

• The first problem we see is integrating fully managed service into CD pipeline? Service can be in private VPC, might not be accessible from build agent, etc.
• In some cases, you can find a distribution of managed service which can be used on premise. But you stumble over another problem immediately: where to host service, how to set it up on build environment, how to keep up to date version etc. The same problem takes place for developer environment, when you want run integration tests locally.
• Next problem is dependencies. Service may require special frameworks, execution environment, libraries etc. You have to deal with their installation. Their versions may collide with existent components used in build environment.
• In some cases, you need prepare test data. It is also non-trivial task. Your data should be immutable between test runs. You need to store them somewhere and have some tool to easy populate them.
• You want your tests to be executed fast

This is where Docker steps in

Docker is free software that provides the ability to package and run an application in a loosely isolated environment called a container. Containers are lightweight and contain everything needed to run the application, so you do not need to rely on what is currently installed on the host. Also, containers solve the “it works on my machine” headache.

All you need to run container is Docker installed on the machine. You can run anything in a Docker container, which makes them ideal for integration testing. Using Docker containers, you can create throwaway instances of anything you need, including your external dependencies. Let’s look a bit deeper into the entire Docker architecture.

How does it help us?

Let’s go back to the example we considered before.

We have dependency represented by the ElasticSearch service (OpenDistro). We want to execute our integration tests against of our API and we want to have OpenDistro working behind the scene (simulate production environment). For that we have to package Java, OpendDistro, Kibana and all its dependencies into Docker container.

Does it sound a bit scary?! Not at all.

Fortunately, we don’t need to do almost anything. AWS already has a packaged version of the ElasticSearch service; all we need just point Docker to run a container with necessary image version.

Just one small part left. How do integrate Docker into CD pipeline?

We need to tell Docker to run the container at a specific time, with specific parameters, like port number needs to be exposed, credentials, some environment variables etc. Fortunately, Docker provides us API to communicate with it.

Which options we have to communicate with Docker?

Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers. The Docker client and daemon can run on the same system, or you can connect a Docker client to a remote Docker daemon. The Docker client and daemon communicate using a REST API, over UNIX sockets or a network interface. Another Docker client is Docker Compose, that lets you work with applications consisting of a set of containers.

So, Docker REST API and Docker Compose are good candidates to be integrated into our CI/CD pipeline for integration testing. Let’s look closer to both of them.

Docker Compose

Compose is a CLI tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

It is how our Docker compose might look to run OpenDistro:

But there are some problems:

• you can’t create and control environment from code if you need (or it can be much harder than with other option)
• your Docker compose file, scripts to run it (or CD steps description) and codebase are separated from each other.

We will not focus on this option because we want more control over the Docker from CD pipeline.

Docker REST API

Besides CLI, Docker has several REST API clients. You can connect to the REST API with the language of your choice and use the corresponding client to do everything you want with Docker (just like you do via cli, but from code).

This makes it easy to automate the Docker setup and deployment and integrate it with test code to create an easy setup of the dev environment, uniform build and test environments (self-contained and portable).

The main Docker REST client for dotnet developers is Docker.Net which is developed by Microsoft. It is fully asynchronous, designed to be a non-blocking and object-oriented way to interact with your Docker daemon programmatically. The client is robust, good and gives you full control over the Docker demon, but we would like to use something even more simple and more declarative approach for our test containers.

Fortunately, there are few libraries/frameworks created over Docker.Net to hide any complexity. All of them provide ready for use container wrappers for most popular cloud services (like Elasticsearch, MsSQL, Mongo, Redis, MariaDB, BigTableEmulator, DatastoreEmulator, InfluxDB etc). Also they provides other features like: creating your own containers, container readiness checks , fluent API support and etc.

First library is Testcontainers. It is port of same name library from Java to .Net platform. Original version has huge amount of ready for use containers and good documentation, but unfortunately .Net version has lack of documentation, slightly behind in feature set from original version and it is hard to figure out what is implemented and what is not. We found many pitfalls when we have start to use it and eventually decided switch to something else.

Second library is .NET Testcontainers. It is completely reworked version of Testcontainers considered above. It has some documentation, and natively written for dotnet and provides cool fluent API. But we had no chance to use it in practice, so leave you experience in comments under the article.

Next library is TestEnvironment.Docker and this is our choice.

TestEnvironment.Docker

It is actively developed and natively designed for dotnet infrastructure, it has some documentation, samples and extensibility points. Unfortunately it has no OpenDistro container implementation out of the box, but we can easy implement it (the code below is just for demonstration purpose and you should never hardcode username and password in source code).

There are two more components we have to implement. The first, container readiness check. This component tell us that container up and running and we can start using it.

The second, container data cleaner. This component clean ups container data if there are some of them.

So, we are almost ready. We have container wrapper, which can run our dependency (OpenDistro) from a code at time when we need it. For our integration tests we will use XUnit framework, so let’s write a fixture that will set up our virtual environment:

Some remarks about the code above:

• we map OpenDistro port to 9201 because we don’t want collide with default port in case we have ElasticSearch installed locally
• we don’t want destroy container in Debug mode, when we running integration tests on local machine. It allows us connect to container in case of some errors and check in details what happened
• we using HostBuilder and in-memory configuration provider to provide other components information how virtual environment was configured (e.g. OpendDisrto url)

What else can we do in this fixture? Well, we can restore index snapshot for OpenDistro or we can prefill indices with necessary data. We can create read-only fixture to prevent any changes in index data during integration tests run, or we can create write-only fixture to cover with integration tests our ETL process. But for now we will leave this simple.

This is how our integration test might look with OpenDistro Docker container which is ran by our fixture behind the scene:

To run our integration tests we can write something like this:

dotnet test src/Tests/MyProject.Tests.csproj — filter Category=Integration

And that’s it!!!

With this approach you can package into container and run within integration tests pipeline almost whatever you want, e.g. we can cover with integration tests ETL part of our system (putting MS SQL server into container).

So, I hope you enjoyed this post and it was useful to you, as well as source code you can look at for some details.

We are ACTUM Digital and this piece was written by Mikhail, Senior .NET Developer of Apollo Division.

If you seek for help with anything related to software testing or cloud setup, just drop us a line. 🚀