Quality Gate: Configuring Code Coverage Checks for .Net Core Microservices in Azure DevOps

How to check that a commit with new code is covered by tests? How to control that tests will be written? What type of coverage metric to choose for analysis?

You can use a simple approach, set up Build Validation, the check will start with each Pull Request to the branch selected for validation and include Code Coverage verification steps, stopping the build if the coverage percentage has decreased. This approach is called “Quality Gate”.

Quality Gate is a enforced measure or metric built into the Pipeline that the software must meet before it can move on to the next step. This measure enforces certain rules, metrics, or practices that code must follow to prevent poor quality code from infiltrating the software being developed.

Quality Gate can be easily configured in Azure Pipelines. The build and coverage analysis steps discussed in the article are universal and can be used in any CI/CD system, such as Jenkins, TeamCity, etc.

In order to easily test the solution you will need:

Tutorial project: https://github.com/devops-stack/eShopOnContainers

Create an account in Visual Studio or sign in with a GitHub account: https://azure.microsoft.com/en-us/products/devops/pipelines

Preparing and launching a project, a simple practical example

Preparing and For clarity, it is best to start with a practical example. In the free subscription, you can clone the repository and create your own pipeline, the process of creating and running is simple and takes 3 minutes. The free Azure DevOps (VisualStudio) subscription has a build agent available so you can quickly see all yourself:

1. Sign up or sign in with a GitHub account (the registration process is as easy as possible): https://azure.microsoft.com/en-us/products/devops/pipelines and click the “Start Free” button.
2. Then click “+ New Project”, enter a project name in the “Project Name” field and click “Create”. Once created, go to the “Repos” section, in the “Import a repository” section click “Import”, copy the link https://github.com/devops-stack/eShopOnContainers into the “Clone URL” field and click “Import”.
3. After cloning, go to the “Pipelines” section, click the “Create Pipeline” button, select “Azure Repos Git”, then select the repository you just created and click on “Existing Azure Pipelines YAML file”. Select “main” branch, copy path to pipeline “src/Services/Catalog/azure-pipelines.yml” then “Continue” and press “Run”. Your build is ready!

You can also just go and see the configured pipeline and code (you need a Visual Studio or GitHub account, the registration process is as simple as possible).

PROD Grade level project example

The article uses a project developed by the Microsoft team, it is a ready-to-use online store developed on .Net Core, using a microservice architecture and following design and development standards. Unit Tests and Functional Tests have also been added, so the store is convenient to use for demonstrations and practical examples (I tried to tell more about this development). The project is conveniently deployed locally, in Kubernetes, Docker Compose, or Azure Kubernetes Service. The repository also contains several books that fully cover the development and operation cycle, which are very convenient for studying technology from different angles of QA, Development, DevOps. The article will use the code and tests of the “Catalog” service of this online store.

To simplify the example and for clarity, only one of the project services will be tested, and the pipeline uses direct paths to the project files, while variables are more often used in real environments. To analyse the coverage of a repository with a large number of services, the process is slightly more complicated. The second article will consider a universal template for different services using variables.

Pipeline steps with Code Coverage and Build Quality Checks

Preparing and checking the coverage consists of a few simple steps, you can see the code of the fully finished pipeline at the link in the repository fork. The following is a description of the steps with a short comments.

Tasks #1 and #2: Restore and Build

The first two tasks update dependencies and build the service (the entire article uses the main branch):

  - task: DotNetCoreCLI@2
    displayName: Restore
    inputs:
      command: restore
      projects: 'src/Services/Catalog/Catalog.API/Catalog.API.csproj'
      nugetConfigPath: 'src/NuGet.config'
  
  - task: DotNetCoreCLI@2
    displayName: Build
    inputs:
      command: build
      projects: 'src/Services/Catalog/Catalog.API/Catalog.API.csproj'
      arguments: '--configuration debug'

Task #3 Run tests and build data collection

To collect data on the amount of tested code, we will use the Coverlet collector using the --collect "XPlat Code Coverage"parameter. This key is used for Linux environments. This is a cross-platform option based on the .NET CLI, which is great for build systems where MSBuild is not available. In a Windows environment, you can use the --collect "Code Coverage" option. Cobertura will be used to calculate the percentage of code that tests access.

 - task: DotNetCoreCLI@2
    displayName: 'Execute unit tests'
    inputs:
      command: 'test'
      arguments: '--configuration debug --collect:"XPlat Code Coverage" \
        -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
      publishTestResults: true
      projects: 'src/Services/Catalog/Catalog.UnitTests/Catalog.UnitTests.csproj'

To successfully complete this step, the following dependency must be added to the csproj file:

<PackageReference Include="coverlet.collector" Version="6.0.0" />

Task #4 Publication of collected statistics

The “Publish Code Coverage Results” step is required for the final step, in order to prepare the artefact in the Cobertura format and mark it up in the desired location.

    displayName: 'Publish code coverage report'
    inputs:
      codeCoverageTool: 'Cobertura'
      summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'

After completing steps 3 and 4, the “Tests” and “Code Coverage” tabs will become available.

The tabs display information about the execution of tests and detailed “Code Coverage” statistics, respectively:

Task #5 Setting up Build Quality Checks

The “Build Quality Checks” step allows you to add a “Quality Gate” to the pipeline. Like the previous steps, this step is quite simple, but it is the icing on the cake, we will use the Azure DevOps branch policy to configure the build to stop if the coverage threshold is lowered. This will protect the project from reducing the level of code coverage with tests. “Build Quality Checks” at the first start analyses the code coverage and generates Baseline — the base value from which the reduction in code coverage will be calculated, this is simply the percentage of coverage of the previous successful build. You can also set the desired coverage percentage.

  - task: BuildQualityChecks@9
    displayName: 'Check build quality'
    inputs:
      checkCoverage: true
      coverageType: lines

An example with a fixed coverage percentage set manually:

steps:
- task: BuildQualityChecks@9
  displayName: 'Check build quality'
  inputs:
    checkCoverage: true
    coverageFailOption: fixed
    coverageThreshold: 70

Сode Сoverage and Branch Coverage

The two most popular coverage metrics are code coverage and branch coverage. In the final step “Build Quality Checks” analysis with Code Сoverage metric is selected by default. It is easier to start the implementation of the metric with control over the decrease in the percentage of coverage compared to the previous build and develop an appropriate percentage based on the experience of developing project requirements.

If you have just started working with metrics, it is better to start from a simple one, build by build, analyse how exactly the percentage of code coverage of your project is formed and not make strict coverage requirements when.

Branch coverage provides a deeper analysis than code coverage. Instead of using the number of lines of code, this metric focuses on structures like “if” and “switch” statements and makes test development a little more difficult.

Visual Studio Test evaluates how many lines of code as well as how many blocks of code have been covered by the tests. Other tools such as Cobertura use other types of coverage such as Branch Coverage which is similar to Block Coverage.

Quality Gate: Setting the Azure DevOps branch policy

Let’s do a simple setup for Quality Gate in Azure Pipelines:

1. In Azure Pipelines, go to “Repos” -> “Branches”, hover over the “main” branch, and on the far right, select the three-dot button that appears and then “Branch Policies”.
2. In the “Build Validation” section, click “+”, check that the pipeline that we created at the beginning of the article is selected in the field and click “save”.

Let’s check it:

1. Create a new branch from main, under “repos” > “Branches” click “New Branch”, give the branch a name and make sure “main” is selected in the “Based on” field.
2. In the “Branches” section, go to the created branch (use the search), open the file “/src/Services/Catalog/Catalog.UnitTests/Application/CatalogControllerTest.cs” and delete all the code in the “Get_catalog_items_success()” method, the method itself and two brackets {} leave (see screenshot below).
3. Then click “Commit” > “Commit” and the “Create a pull request” button that appears. Fill in the “Title” field and click “Create”

The main branch is now protected by the pipeline we have developed. When you try to make a PR to a branch, the pipeline will start and in case of a decrease in the percentage of code coverage, it will stop with a message that the percentage of code coverage has decreased relative to the last check. And the configured branch policy will not allow the pull request to be completed.

The execution of a Pull Request does not have to be “stopped”. If you are just building the requirements of a testing approach, it is convenient to set up an optional check and not stop the build. Also in Azure DevOps it is possible to grant rights to change the branch policy to certain people. Thus, when the approach to testing, metrics and the percentage of coverage will be developed, if there is a need, it is possible to strictly regulate the requirements for test coverage.

The next article will add an important step in setting up a template pipeline for a repository with a large number of services. This is necessary in case your platform or web application services are developed by different teams. Then the Quality Gate pipeline should not block the development of other services.

Additional resources:

Excellent book: “Unit Testing Principles, Practices, and Patterns: Effective testing styles, patterns, and reliable automation for unit testing, mocking, and integration testing with examples in C#”

Build, test, and deploy .NET Core apps

Quality Gates in Testing — Powerful but underused by Gipil Quddus

Configuring Test Reports and Code Coverage: https://www.jetbrains.com/help/teamcity/configuring-test-reports-and-code-coverage.html

Jenkins Code Coverage API: https://plugins.jenkins.io/code-coverage-api/

Build Quality Checks repo and readme: https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/overview.md

Coverlet is a cross platform code coverage framework for .NET