Running your unit tests with Visual Studio Team Services and Docker Compose
This is a follow up to my post about Building a multi Docker image solution using Visual Studio Team Services and Docker Compose.
In this post I will explore how to execute simple unit tests in Visual Studio Team Services using Docker Compose. I will also try to collect the test results so Visual Studio Team Services can show you a nice report of successful and failed tests. Last I will reflect on my findings and give my cautious recommendation.
I will cover executing more complex integration tests in another post following this one.
In the previous post, mentioned above, I linked to the sample code I’m using setting up the whole thing. You can find the code on GitHub.
In the sample, the unit tests are located in the “VstsDockerBuild.Tests” project and contain a handful of almost empty xunit tests.
To run the tests I added a simple Dockerfile that will copy in the solution content and run
dotnet restore. It only does the restore part as the
dotnet test command will automatically build the solution. The Dockerfile looks like this:
FROM microsoft/dotnet:2.0-sdk-jessie AS build
COPY . ./app
RUN dotnet restore
This will result in a Docker image that we can use in Docker Compose to actually run the unit tests. As this is a single image/container thing, we could probably run the tests using a regular
docker run command, but Docker Compose provides a very nice and declarative way of running the tests.
In addition to the Docker image we need a Docker Compose file to define how we want to run the tests. This could look something like this:
Now a few interesting things are going on here besides the regular
image parts of the compose file.
First of all I define an environment variable that I can use to force a test failure. Nice when trying to get this running :) The environment variable can be set when queuing a new build using VSTS. Docker Compose will pick it up and set it for the container. See Environment variables in Compose for details.
To run the tests I use the provided Docker Compose task in VSTS that I also use to build the images. I’m using the “Run a Docker Compose command” action and provide the following command
up --abort-on-container-exit. The “abort” argument will trigger when the
dotnet test process exists and it will propagate the exit code from the test process.
dotnet test will exit with
0 on success and
1 on failure.
Returning to the
docker-compose.tests.yml file above, the entrypoint is set to
dotnet test --logger trx --results-directory /var/temp. I set the logger and results output directory in addition to the
dotnet test command. This will generate a .trx file that VSTS can pick up and render a nice test report.
For the test result output to work I also needed to define a volume in the compose file that maps to a temporary folder on the build agent. This is necessary as VSTS can’t get to files inside a Docker container by itself. So using a volume allows for this to work. A last step is to setup the “Publish Test Results” task and provide a few configuration settings.
I tell it to pick up all .trx files in the
/opt/vsts/work/_temp directory that was mapped to the
/var/temp directory in the container.
That is it, when running a build the unit tests will also be executed and we will get a nice test report, hopefully everything will be green :)
Ah! no such luck, it failed. As you can see the “Run integration tests” and “Push images” tasks have been skipped, but the “Publish Test Results” task did run. This behavior can be configured for all tasks by modifying the “Control options”. I have selected to run this task even if a previous task have failed for the “Publish Test Results” task.
While I think running the unit tests in the sample solution using Docker provides a nice insight into how Visual Studio Team Services works and what it provides, I also acknowledge that, for simpler setups, running true unit tests using Docker, might be overkill. I do not think it provides a huge amount of value, especially if your solution is primarily based on .NET or .NET Core where Visual Studio Team Services provide a lot of tools out of the box that will run your tests and provide the result report that I had to jump through a few hoops to get working.
Going forward I don’t think running unit tests using Docker will be my immediate go to solution, especially if the tests are close to, or actually are, true unit tests that can be executed without dependencies on external services etc.
The next step will be to get the integration tests running and getting a feel for how you can run complex tests of an entire application stack. This is where I think, and hope, Docker Compose and Visual Studio Team Services will really shine.