Collecting test coverage using Coverlet and SonarQube for a .net core project

SonarQube is a code quality measuring tool that helps developers to keep an eye on the evolution of their codebase. The best part, to me, is that it comes in form of a Docker Image! This makes running a SonarQube server as easy as :

docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 sonarqube

After the initial setup, it should be good to go.

Now let’s see how we can get our codebase integrated.

The ‘code under test’

First, we’ll need to have a piece of code that represents logic which should be submitted to unit testing. I’ve created a repository called testcoverage and created a subfolder calculation to hold our .net core class library.

In the calculation folder, run the following scaffolding command:

dotnet new classlib

This will generate a C# class library netstandard2.0 project, here we can write our logic.

We’re creating a simple calculation-style logic program that respects the following interface :

public interface ICalculation
{
    decimal Calculate();
}

The implementations for addition, division, multiplication, and subtraction can be written like the following :

The full example for this project can be found on GitHub: https://github.com/merken/testcoverage/tree/master/calculation

Testing the code

With our codebase implemented, we can create a XUnit test project to cover the code with tests. For this, create a folder calculation.tests in the testcoverage repository.

In the calculation.tests directory, run the following scaffolding command:

dotnet new xunit

For this project, we require some external nuget packages. Install Coverlet and FluentAssertions for the calculation.tests project using the following CLI commands:

dotnet add package coverlet.msbuild --version 2.0.1
dotnet add package FluentAssertions --version 5.0.0

Before you can test our logic project, a reference to that project is required, in Visual Studio this is quite easy to do, but I’m focussing on the dotnet CLI, so here’s the command to get this linked to the calculation.tests project:

dotnet add reference ..\calculation\calculation.csproj

Now we can start writing our tests.

TIP: You should practice TDD, I try writing my tests first and seeing where stuff breaks and write the logic afterward.

The following sample tests all four calculation operations using FluentAssertions:

XUnit provides two ways to write test [Fact] and [Theory]. A Fact is a test that has only one result. A Theory can have multiple test-paths and can have multiple results.

A Theory is also a great way to deduplicate your tests using different input parameters. This can be done by providing [InlineData].

TIP : Regarding the Division test, I’ve tried using float for the data type, but it doesn’t throw a DivideByZeroException. Instead, it returns positive infinity. It appears only ints and decimals throw exceptions when dividing by zero.

To prepare our codebase for test coverage and SonarQube, we need a way to link all our projects together in a main entry point that builds all our projects with respect to their references and dependencies. This is known as a solution file, and it is known concept in the .NET world. For .net core projects, this is not always common, but it is supported by the CLI to scaffold such a file:

At the repository level (testcoverage) run the following command:

dotnet new sln

It will inherit the name of the parent folder and create the testcoverage.sln file. Now we need to add our projects to this file.

dotnet sln add .\calculation\calculation.csproj
dotnet sln add .\calculation.tests\calculation.tests.csproj

With this file in place, we can run the dotnet build command at the root of our repository to build all linked projects in one go.

dotnet build

You can also run the dotnet test command to execute the tests, it will try to test the calculation project and fail because it is not a test project. I was hoping the solution file would provide sufficient information to the CLI about the project type, but alas.

Calculating coverage

Coverlet is an open source .net core code coverage library, it can output results in the OpenCover format. Which is superb, because SonarQube can handle this format.

More about Coverlet can be found in this article :

Setting up Coveralls with Coverlet for a .NET Core project

To get coverlet to collect code coverage for our codebase we need just to run the following command at the repository root :

dotnet test calculation.tests/calculation.tests.csproj /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

The results will be collected in a coverage.coverlet.xml file in the calculation.tests directory. To visualize these results, we either need a report generator or tool like SonarQube.

We’ll focus on SonarQube, for this we need a SonarQube instance, which we started in a Docker container at the beginning of the article, and the sonar scanner tool for MSBuild.

To install it globally on your system, you should run the following command:

dotnet tool install --global dotnet-sonarscanner --version 4.3.1

Now we have everything in place to start collecting code coverage and visualizing it in SonarQube. I’ve distilled the process down to a single ‘.cmd’ file :

dotnet test calculation.tests/calculation.tests.csproj /p:CollectCoverage=true /p:CoverletOutputFormat=opencover
dotnet build-server shutdown
dotnet sonarscanner begin /k:"calculation" /d:sonar.host.url=http://localhost:9000 /d:sonar.cs.opencover.reportsPaths="calculation.tests\coverage.opencover.xml" /d:sonar.coverage.exclusions="**Tests*.cs"
dotnet build
dotnet sonarscanner end

First, we calculate the test coverage and output a coverage.coverlet.xml file.

Line 02 shuts down any remaining background build services, just in case any keep a lock on the .sonarqube directory.

Line 03 starts the sonar scanner and configures it to send results to our local instance in Docker with the coverage xml file. All *Tests* files are ignored for brevity. These exclusion rules must be tailored for your solution. Typically you’ll exclude the entire *Test* directory. Keep in mind this is case-sensitive.

Line 04 builds the entire sln (solution) file, all our projects are built.

Finally we end the scanner and the results are sent over to our local sonar instance.

Typically, if a project does not exist, the scanner creates the project based on the key ‘calculation’. If not, you should do this manually.

The results should be shown in SonarQube.

Look at that, a 100% code coverage. This is correct since we test every class from the calculation project in the test project.

This brings me to next topic.

What is good code coverage ?

Simply put, 100% should be your goal eventually, it is not impossible. But it is not always feasible. In a typical frontend-backend setup, we tend to highly invest on covering as much code in the backend as possible, since this is where the most of the business logic resides and use a end-to-end testing tool like Cypress to cover the frontend happy paths.

Keep your tests useful, if a bug arises, fix it, write tests for it. Make sure it never happens again. Practice TDD to train yourself in writing more tests, pair programming is also highly advised, since it allows the co-pilot to play devils advocate and challenge the driver to write at least a single test before proceeding.

Happy testing.

merken/testcoverage