.NET Core Code Quality with Coverlet and SonarQube, Part 2

By Sandeep Anantha

In a previous blog, I introduced SonarQube, a tool that can identify code smells, bugs, and vulnerabilities. SonarQube can increase .NET Core code quality, especially when used with Coverlet. In this blog, I show how developers can put these tools in place to improve quality.

Source Code Scanning using SonarQube (SQ)

Prerequisites

• .NET Core 2.0
• XUnit
• Docker
• VS Code
• Coverlet
• SonarQube

All the above tools are very popular and need no introduction except for Coverlet and SonarQube.

Coverlet is a cross-platform code coverage tool for .NET Core. As per the official documentation, Coverlet generates code coverage information by going through the following process:

1. Before Tests Run

• Locates the unit test assembly and selects all the referenced assemblies that have PDBs.
• Instruments the selected assemblies by inserting code to record sequence point hits to a temporary file.

2. After Tests Run

• Restores the original non-instrumented assembly files.
• Reads the recorded hits information from the temporary file.
• Generates the coverage results from the hits information and writes it to a file.

For more information on Coverlet, please refer to the official documentation.

You can choose to generate reports in a wide variety of formats such as opencover, lcov, json, and cuberta — opencover is the most popular.

Install Coverlet either globally across the projects or locally scoped to a project (using package reference) using the following commands:

dotnet tool install — global coverlet.console. //global

dotnet add package coverlet.msbuild //local

To generate coverage reports, you can use Coverlet directly by executing the following command:

Syntax:

coverlet <ASSEMBLY> — target <TARGET> — targetargs <TARGETARGS> — format opencover

Example:

coverlet /path/to/test-assembly.dll — target “dotnet” — targetargs “test /path/to/test-project” — no-build — format opencover

SonarQube is a web service that can be installed on a central server so that it is available across the enterprise or department. If SonarQube is not installed and is not available, we can use Docker to set up a local instance of it.

Sample Project

For the purpose of this blog, we will create a class called SimpleMath in a class library, and we will also create tests around the SimpleMath class in a separate test project. Add a SimpleMath class with add, subtract, and divide methods to the Math class library, and add a test class to the test library.

1. Create Parent Folder

mkdir Math
cd Math

2. Create Math Project

mkdir Math
cd Math
dotnet new classlib

Add a simple class that has basic math operations as methods. You can pull down the sample code here.

Note: There is a condition in the divide method (DividebyZero) that is executed only when the second parameter is 0. If you never call this method passing 0 for the second parameter from the test class, the coverage will never be 100%.

Note: The multiply method is intentionally commented out to demonstrate the coverage metric.

3. Create Test Project for Math

cd ..
mkdir Math.Test
cd Math.Test
dotnet new xunit

4. Add a Reference to the Above Math Project

dotnet add reference ../Math/Math.csproj

5) Add a Test File with the Code Below That Tests the Math Methods Created Above.

using System;
using Math.Api;
using Xunit;

6. Create the Solution File

cd ..
dotnet new sln
dotnet sln add Math/Math.csproj
dotnet sln add Math.Test/Math.Test.csproj

7. Create Coverage Reports

Once the code base is ready, and the corresponding test project is created and built, it is time to generate coverage reports off of the test project.

Running Coverlet

With setup out of the way, let’s dive into Coverlet. We can integrate coverlet right into msbuild by installing these 2 packages to your test project.

cd Math.Test
dotnet add package coverlet.msbuild — version 2.0.1
dotnet add package FluentAssertions — version 5.0.0

Now, msbuild will generate coverage reports when you pass the right options.

cd ..
dotnet test Math.Test/Math.Test.csproj /p:CollectCoverage=true \
/p:CoverletOutputFormat=opencover

Once the build runs, you should see the below output on the console. In addition to this, a file(‘coverage.opencover.xml’) will also be created in the test folder that has a coverage report. I used the opencover format which is very popular, but you can also use various other formats. Refer to the Coverlet documentation for more information.

You can clearly see that coverage is 100%. But to test it, if you comment out the DivideTest_ThrowsException case that covers the divide by zero case, and run it again, you will see the coverage drop by about 10%. This is happening as the Dividebyzero branch code is not tested by any method and is not covered, hence Coverlet marks it as not covered.

You can clearly see that Coverlet is using tests to verify the coverage of the code.

Integration with SonarQube

Now that coverage reports have been generated, we have to send these reports to SonarQube for further analysis of code smells and vulnerabilities. For this demo, we will use a local instance of SonarQube. Since installing SonarQube is a tedious process, we will set up a local Docker instance of SonarQube by running the command below:

docker run -d — name sonarqube -p 9000:9000 -p 9092:9092 sonarqube

P.S.: If you run into a port conflict, please change the host port number (9000) to something else, but the container port number should remain the same

We need to install the SonarScanner tool to analyze .NET assemblies.

dotnet tool install — global dotnet-sonarscanner

After it is installed, execute the below commands to start SonarScanner and to send a report result to SonarQube:

a) dotnet test Math.Test/Math.Test.csproj /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

b) dotnet build-server shutdown

c) dotnet sonarscanner begin /k:Math /d:sonar.host.url=http://localhost:9000 \
/d:sonar.cs.opencover.reportsPaths=Math.Test/coverage.opencover.xml \
/d:sonar.coverage.exclusions=”**Test*.cs”

d) dotnet build

e) dotnet sonarscanner end

Here is a short description of the above commands:

• Since Coverlet is integrated with msbuild, Coverlet switch parameters are passed in the dotnet test command directly, which are then passed down to Coverlet. So the .NET test will piggyback on Coverlet to generate coverage reports. This command will generate coverage.opencover.xml in Math.Test folder.
• Shutdown all msbuild tasks to not interfere with these project folders.
• As mentioned, SonarScanner feeds off of coverage reports, so we have to pass path as a parameter. This command starts the scan process and sends results to the local instance of the SonarQube. We also have to exclude the tests project as we do not want to measure coverage on tests itself.
• Build the solution. This will create a SonarQube folder and will save analysis reports.
• End the SonarScanner process

Voila, that’s it. Browse the SonarQube dashboard (http//localhost:9000/) and you should see the coverage results.

1. Browse http://localhost:9000/projects
2. You should see Math project under projects and coverage as 100%
3. Quality gate for the project should be passed

You can make changes to the code, repeat the process, and see it reflected in the SonarQube dashboard.

Quality Gates

An additional feature of SonarQube is its ability to mark a build as pass or fail based on the quality gate and SonarQube comes with default quality gates for popular languages. Quality gates in SonarQube are a set of rules used to enforce quality standards. If the code has not met the defined rule, SonarQube will mark it as a fail. We can also customize these rules per organizational divisions and business unit.

For details on SonarQube quality gate, click on the quality gates tab (http://localhost:9000/quality_gates/show/1)

Take a look at the definition of a default quality gate. One of the rules is that “Coverage on New Code” should be greater than 80%. What this means is that if we add new code without adding corresponding tests then the coverage for new code drops below 80%. Hence, it will be treated as an error and the quality gate will fail.

To induce this error, let’s add a new multiply method (or uncomment if you have the same sample code we started with) to our SimpleMath class without adding any tests to it.

public decimal multiply (int a, int b) {
return a * b;
}

If you repeat the same process again you will see overall coverage drop by 20%. But the important thing to note is that coverage on new code is less than 80%. The SonarQube quality gate fails with a red flag.

Clicking on the Math Project link, we get into the details. Here you can clearly see that the quality gate fails as the coverage is below 80%.

If you add tests for the ‘multiply’ method as well, this Quality Gate will pass. This is how SonarQube enforces quality standards.

In addition to this, SonarQube will also fail in case the reliability/maintainability or security rating falls below A.

Quality Gates can be integrated into automated CI/CD (Jenkins/Concourse/Cruise control) and deployment can be stopped if it fails. If there is no off-the-shelf resource to query quality gate status, SonarQube also exposes this status over REST.

P.S.: Source code for this demo can be cloned from this link.

Sandeep Anantha is an Agile software engineer at TribalScale.

TribalScale is a global innovation firm that helps enterprises adapt and thrive in the digital era. We transform teams and processes, build best-in-class digital products, and create disruptive startups. Learn more about us on our website. Connect with us on Twitter, LinkedIn & Facebook!