.NET Code coverage on GitLab

Testing our software is a crucial development stage to maintain a high standard of quality. To reduce the amount of bugs being released into the production environment, code coverage, along with other tests, can be used as a helpful measurement index.

Code coverage is a measure that describes how much of a software source code is executed when a particular test suite runs. High test coverage means that the code has a lower chance of containing undetected bugs compared to a program with lower outcomes.

There are many different metrics that can be used to calculate test coverage; some of the most basic are the percentage of lines and branches called during execution of the test suite.

To monitor it, GitLab, through its CI/CD pipelines, offers the possibility to run your tests, parse the coverage percentage from the stdout through a regular expression and show it on merge requests.

Unfortunately, this isn’t the case with .NET projects, since there is no built-in support from Gitlab to .NET, but in TUI Musement we care about quality, so we found a solution!

NOTE: you should not rely on coverage percentage alone, tests must be properly written to cover any corner case your code could face!

Get started

In order to produce the necessary artifacts, we add 2 packages to the test project: Coverlet, a cross platform framework with support for line, branch and method coverage, and Junit, a tool that allows to output the XML format GitLab needs.

dotnet add package coverlet.collector
dotnet add package JunitXml.TestLogger

You can find a sample project here.

Run tests

First, we need to make sure to run our tests properly and collect the coverage data, to do that we use the following command paying particular attention to the —-collect and —-logger options:

dotnet test -c Debug
            -r ./cobertura
            --collect:"XPlat Code Coverage"
            --test-adapter-path:.
            --logger:"junit;LogFilePath=./junit/junit-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"

This produces the following files containing all the needed information.

cobertura/
  {auto generated hash}/
    coverage.cobertura.xml
...
tests/
  Weather.Tests/
    junit/
       junit-test-result.xml

It’s time to automate this process! Add a job to .gitlab-ci.yml in order to get the work done for us and use the generated artifacts automatically every time the pipeline is triggered according to the defined rules. In this case, to keep it simple, they have been omitted.

test-with-coverage:
  image: mcr.microsoft.com/dotnet/sdk:5.0-alpine
  stage: tests
  variables:
    CONFIGURATION: "Debug"
    COVERAGE_FLAG: "XPlat Code Coverage"
    LOGGER_FLAG: "junit;LogFilePath=$CI_PROJECT_DIR/junit/junit-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"
  script:
    - 'dotnet test
            -c $CONFIGURATION
            -r $CI_PROJECT_DIR/cobertura
            --collect:"$COVERAGE_FLAG"
            --test-adapter-path:.
            --logger:"$LOGGER_FLAG"'
  artifacts:
    paths:
      - $CI_PROJECT_DIR/cobertura/*/coverage.cobertura.xml
      - $CI_PROJECT_DIR/junit/junit-test-result.xml
    reports:
      cobertura:
        - $CI_PROJECT_DIR/cobertura/*/coverage.cobertura.xml
      junit:
        - $CI_PROJECT_DIR/junit/junit-test-result.xml

Using the mcr.microsoft.com/dotnet/sdk:X.X-alpine Docker image to execute the dotnet test command, we’re telling GitLab to retrieve the XML files through the paths keyword (it’s important to use the wildcard * since the folder name is an auto generated hash) and use them as reports through the homonymous keyword.This allows GitLab to show the covered lines directly into the merge request, but, since the dotnet test command doesn’t print it to stdout, the total coverage percentage is still missing.

Retrieve the coverage percentage

Since the test command doesn’t produce any output concerning the total coverage percentage, we need to extract it from the newly created coverage.cobertura.xml report and print it to stdout using the following script:

#!/usr/bin/env shREPORTS_DIR="${1}"if [[ $# -lt 1 ]]; then
    echo >&2 "You have to provide the directory containing your coverage reports."
    exit 2
fitotal_coverage=0
count=0for i in $(find "$REPORTS_DIR" -name '*.xml');
do
    printf "Found coverage report: %s\n" "$i"
    line_rate="$(head -n 2 ${i} | sed 'N;s/.*line-rate="\([^" ]*\).*/\1/g')"
    coverage=$(echo "${line_rate} * 100" | bc)    printf "PARTIAL_COVERAGE: %2.2f\n" "$coverage"    count=$((count + 1))
    total_coverage=$(echo "${total_coverage} + ${coverage}" | bc)
done;printf "Found a total of %i report(s)\n" "$count"
printf "TOTAL_COVERAGE=%2.2f\n" "$(echo "${total_coverage} / ${count}" | bc -l)"

Update the test-with-coverage job to make it execute the script by adding these new lines under the script keyword:

- chmod +x ./scripts/print-dotnet-coverage.sh
- ./scripts/print-dotnet-coverage.sh $CI_PROJECT_DIR/cobertura

Define the regular expression that will parse the stdout to get the desired value by adding this new keyword to the job:

coverage: /TOTAL_COVERAGE=(\d+.\d+)/

This will finally allow us to see the coverage percentage directly into the merge request.

Publish the report

Now, in order to make the report easily available for consultation, we generate an html version using ReportGenerator and publish it to a GitLab environment introducing the following job:

deploy-coverage-report:
  image: mcr.microsoft.com/dotnet/sdk:5.0-alpine
  stage: deploy artifacts
  needs:
    - job: test-with-coverage
      artifacts: true
  variables:
    PUBLIC_URL: "/-/$CI_PROJECT_NAME/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public"
  environment:
    name: Coverage/MyWeatherAPI
    url: "https://$CI_PROJECT_ROOT_NAMESPACE.gitlab.io/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public/index.html"
  script:
    - dotnet tool install --tool-path . dotnet-reportgenerator-globaltool
    - mkdir -p public
    - ./reportgenerator "-reports:$CI_PROJECT_DIR/cobertura/*/coverage.cobertura.xml" "-targetdir:public" "-reporttypes:Html"
  artifacts:
    paths:
      - public

In this job we’re inheriting the artifacts from test-with-coverage, using the aforementioned tool to generate the html report and telling GitLab to publish the result to an environment named MyWeatherApi under the Coverage folder.

After the next successful run, you can go to Deployments > Environments, select the right one and click View deployment to reach your up-to-date coverage report and explore it.
A link will also be directly available into every interested merge request.

Now .NET projects can finally benefit from all GitLab code coverage features and, thanks to the deployed environment, you can analyze the report whenever you want to make sure all values are stable or decide if improvements are needed.

Useful links:

https://docs.gitlab.com/ee/ci
https://github.com/coverlet-coverage/coverlet
https://github.com/spekt/junit.testlogger
https://github.com/danielpalme/ReportGenerator