Coverlet MSBuild Integration -Merging reports

Coverlet is a cross platform code coverage framework for .NET, with support for line, branch and method coverage. It works with .NET Framework on Windows and .NET Core on all supported platforms.

A few words about code coverage

Code coverage is a software quality metric, showing how much of the code is exercised when your tests run. Code coverage doesn’t talk about the quality of the tests, but it does give us some understanding about our tests. For example, when our method has lots of branches (condition statements), the metrics show us how many of those branches were exercised in the tests.
Code coverage doesn't focus only on unit tests, but all kind of testing(even manual). Anything that involves running the code is a valid way to get the code coverage metrics.

There are many tools to get the code coverage metrics for our unit/integration tests; I would focus on Coverlet.

Coverlet can be used through three different drivers:

• VSTest engine integration
• MSBuild task integration
• As a .NET Global tool (supports standalone integration tests)

At the moment VSTest integration doesn’t support all features of msbuild and .NET tool, for instance show result on console, report merging and threshold validation. We’re working to fill the gaps. (from coverlet github)

MSBuild Integration:

I created two class libraries: MathFunctions.csproj, SimpleArithmetics.csproj. In addition I created two tests projects, each references one of the class libraries it tests: MathFunctionsTests.csproj, SimpleArithmeticsTests.csproj.

Install coverlet package on each of the tests projects:

dotnet add package coverlet.msbuild

Csproj files should contain the following:

<ItemGroup>
<PackageReference Include="coverlet.msbuild" Version="3.1.2">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.1.0" /><PackageReference Include="MSTest.TestFramework" Version="2.1.0" /></ItemGroup>

From sln location run the following commands:

dotnet test SimpleArithmeticsTests\SimpleArithmeticsTests.csproj /p:CollectCoverage=true  /p:CoverletOutput=../CoverageResults/
dotnet test MathFunctionsTests\MathFunctionsTests.csproj /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json"

This will create a coverage.json file that contains the code coverage report combining both tests project execution.

You can also run dotnet test and merge with single command, and generate a valid format like cobertura, but you need to ensure that tests will run sequentially(-m:1). This slows down testing, but avoids invalid coverage result:

dotnet test /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/ /p:MergeWith=”../CoverageResults/coverage.json” /p:CoverletOutputFormat=”cobertura” -m:1

This command will create a coverage.cobertura.xml file that contains the code coverage report combining both tests project execution.

If you are running your tests on azure devops, I would recommend using cobertura as CoverletOutputFormat. Azure DevOps provides us with the ability to show an overview of the code coverage of our application’s unit tests that are generated in a cobertura format.