Test Coverage, Code Scan, Dependency Scan and Security Scan in an Azure DevOps Pipeline
In this article, I will look at how to calculate test coverage, perform code, dependency and security scanning using an Azure DevOps CI/CD pipeline.
Code coverage is a measurement of how many lines/blocks/arcs of your code are executed while the automated tests are running.
Code coverage is collected by using a specialized tool to instrument the binaries to add tracing calls and run a full set of automated tests against the instrumented product. A good tool will give you not only the percentage of the code that is executed, but also will allow you to drill into the data and see exactly which lines of code were executed during a particular test.
What is code coverage and how do YOU measure it?
Asked What is code coverage and how do YOU measure it? I was asked this question regarding our automating testing code…
Code coverage is important. The real benefit of having unit tests can only be felt after code coverage reaches a good level (typically something over 60%).
Having code coverage computed during builds helps developer to know if they have created enough automated tests.
Set the publishTestResults flag
The first step to compute test coverage is setting the
publishTestResult flag when tests are run. The .NET Core CLI can use Coverlet to compute test coverage.
- task: DotNetCoreCLI@2
displayName: 'Run unit tests - $(buildConfiguration)'
arguments: '--no-build --configuration $(buildConfiguration) /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/'
Compute coverage report
The second step is computing the report starting from the test results. Cobertura is one of the tools available in Azure Pipelines.
- task: DotNetCoreCLI@2
displayName: 'Create code coverage report'
arguments: 'run reportgenerator -reports:$(Build.SourcesDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:HtmlInline_AzurePipelines'
The third step is optional and is about making the report available to developers in a tab in the build result.
Report Generator is an extension that helps with that.
- task: PublishCodeCoverageResults@1
displayName: 'Publish code coverage report'
I recommend reading this Microsoft Learn article to find more details on how to compute code coverage:
Exercise - Perform code coverage testing - Learn
Much like the tool you use for unit testing, the tool you use for code coverage depends on the programming language and…
One of the most popular code scanners available on the market is SonarQube.
I tried their extension for Azure DevOps and I configured my environment by installing the following tools:
- the SonarQube Community edition that comes shipped also as a Docker container
- ngrok to expose my local server URL to the public
- the SonarQube Azure DevOps extension, available for free in the marketplace
Run SonarQube Community Edition
SonarQube Community Edition comes shipped in a container that requires Docker installed on a machine.
Try Out SonarQube
You’ve heard about how SonarQube can help you write cleaner and safer code, and now you’re ready to try it out for…
After the container is up and running, SonarQube will be running on your local machine and will be reachable at
Connect Azure DevOps and SonarQube
Next is to set up a connection between Azure DevOps and SonarQube.
This connection needs to be two-ways: Azure DevOps will need to access SonarQube during a build using a service connection. SonarQube will need to keep track of the metadata associated with repositories.
I followed the steps detailed on this page:
Azure DevOps Integration
SonarQube’s integration with Azure DevOps allows you to maintain code quality and security in your Azure DevOps…
To create a service connection, I needed to make my local SonarQube server available to the internet and I run the following command using ngrok:
ngrok http 9000
Modify the CI/CD Pipeline
SonarQube recommends running these three tasks in a pipeline in order to perform static code analysis:
- task: SonarQubePrepare@4
projectKey: 'AZ402_Space_Game_-_web'- task: SonarQubeAnalyze@4- task: SonarQubePublish@4
I run into some problem running
SonarQubeAnalyze as it requires .NET Core 2.* installed on the build server.
In my scenario a fourth task was also needed:
- task: UseDotNet@2
displayName: 'Use .NET Core SDK 2.2.203'
For Java applications, the Analyse task won’t be required and separate steps will be needed to integrate the Maven or Gradle build.
After these, I could successfully run a pipeline and perform a scan. The results are awesome. They offer a nice break down that helps to assess how the quality of code is affected by a commit and helps to keep track of it.
Worth noticing that using this configuration it took more than 4 minutes for the analysis to be completed:
In the Java world three tools are broadly used.
CheckStyles is a static code analysis tool and checks if code is compliant with specified coding rules.
PMD is as well a static code analyzer that looks fro common programming flaws.
See also Getting Started Download pmd-bin-6.33.0.zip Extract the zip-archive, e.g. to C:\pmd-bin-6.33.0 Add folder…
FindBugs scan code looking for bugs.
Potential errors are classified in four ranks: (i) scariest, (ii) scary, (iii) troubling and (iv) of concern. This is a hint to the developer about their possible impact or severity.
FindBugs - Wikipedia
FindBugs is an open-source static code analyser created by Bill Pugh and David Hovemeyer which detects possible bugs in…
WhiteSource Bolt is a dependency scanner that provides information on the dependencies that are referenced by a project, like imported vulnerabilities or potential licencing issues.
A nice extension for Azure DevOps makes dependency scanning easy to perform:
WhiteSource Bolt — Visual Studio Marketplace
We help you harness the power of open source without compromising on security or agility! WhiteSource Bolt is a FREE…
In order to test it, a commercial licence is required.
The report highlighted a number of vulnerabilities on my mock project:
The report also included information about the dependency’s licences:
And a breakdown of the commercial risk:
OWASP ZAP is an extension that allows performing a dynamic security scan against a running web application.
OWASP ZAP requires a continuous delivery pipeline in place. The extension will pretend to be an attacker and will fiddle the web application with requests to try to identify vulnerabilities.
A great article with a detailed step to step procedure on how to install is available at this link:
OWASP ZAP Scanner: Integrating to Azure DevOps Release Pipeline
Setting up OWASP ZAP Scanner in Azure DevOps release pipeline.
I added a separate stage in my CI/CD pipeline:
- stage: 'SecurityScanning'
displayName: 'Security Scanning'
- job: 'Performscan'
displayName: 'Performing Security Scan'
- task: owaspzap@1
port: '443' - task: PublishBuildArtifacts@1
I managed to run the tool against my mock application with these results:
How Much will this cost?
The next question is, how having these four awesome extensions would cost?
Code coverage and security scan may be both be implemented with open source tools as those that have been discussed in this article.
Costs of a dependency scan with WhiteSource Bolt might vary depending on the features that are needed:
Prices range from $0 to $28,000.
In this article, I tried the Essentials tier.
Regarding SonarQube, I would definitely recommend using at least the developer edition as the community edition can only run on the
The cost will be proportional to the size of the codebase that has to be scanned:
The Pipeline I used
For reference I am also using the pipeline that I used to write this article.
In this article, I looked at how a CI/CD pipeline can have a huge impact on the quality of the code that is written.
Thank you for reading this article, I hope you found it useful.