CODEX

Test Coverage, Code Scan, Dependency Scan and Security Scan in an Azure DevOps Pipeline

In this article, I will look at how to calculate test coverage, perform code, dependency and security scanning using an Azure DevOps CI/CD pipeline.

Alberto De Natale
Mar 28 · 7 min read

Code Coverage

Code coverage is a measurement of how many lines/blocks/arcs of your code are executed while the automated tests are running.

Code coverage is collected by using a specialized tool to instrument the binaries to add tracing calls and run a full set of automated tests against the instrumented product. A good tool will give you not only the percentage of the code that is executed, but also will allow you to drill into the data and see exactly which lines of code were executed during a particular test.

Code coverage is important. The real benefit of having unit tests can only be felt after code coverage reaches a good level (typically something over 60%).

Having code coverage computed during builds helps developer to know if they have created enough automated tests.

Set the publishTestResults flag

The first step to compute test coverage is setting the publishTestResult flag when tests are run. The .NET Core CLI can use Coverlet to compute test coverage.

Compute coverage report

The second step is computing the report starting from the test results. Cobertura is one of the tools available in Azure Pipelines.

Publish results

The third step is optional and is about making the report available to developers in a tab in the build result.

Report Generator is an extension that helps with that.

ReportGenerator creates user-friendly reports starting from different coverage formats.

Results

The report details which line of code are untested.

I recommend reading this Microsoft Learn article to find more details on how to compute code coverage:

Code Scan

One of the most popular code scanners available on the market is SonarQube.

I tried their extension for Azure DevOps and I configured my environment by installing the following tools:

My local setup used ngrok to expose SonarQube to the internet. This should not be used for production.
  • the SonarQube Community edition that comes shipped also as a Docker container
  • ngrok to expose my local server URL to the public
  • the SonarQube Azure DevOps extension, available for free in the marketplace

Run SonarQube Community Edition

SonarQube Community Edition comes shipped in a container that requires Docker installed on a machine.

The SonarQube container is up and running.

After the container is up and running, SonarQube will be running on your local machine and will be reachable at http://localhost:9000

http://localhost:9000

Connect Azure DevOps and SonarQube

Next is to set up a connection between Azure DevOps and SonarQube.

This connection needs to be two-ways: Azure DevOps will need to access SonarQube during a build using a service connection. SonarQube will need to keep track of the metadata associated with repositories.

I followed the steps detailed on this page:

To create a service connection, I needed to make my local SonarQube server available to the internet and I run the following command using ngrok:

Azure DevOps is running requests against my SonarQube local instance.

Modify the CI/CD Pipeline

SonarQube recommends running these three tasks in a pipeline in order to perform static code analysis:

I run into some problem running SonarQubeAnalyze as it requires .NET Core 2.* installed on the build server.

In my scenario a fourth task was also needed:

For Java applications, the Analyse task won’t be required and separate steps will be needed to integrate the Maven or Gradle build.

Result

After these, I could successfully run a pipeline and perform a scan. The results are awesome. They offer a nice break down that helps to assess how the quality of code is affected by a commit and helps to keep track of it.

SonarQube quantified 54 days worth of technical debt for my mock application.

Worth noticing that using this configuration it took more than 4 minutes for the analysis to be completed:

The scan took some time, probably because of my weird setup.

Java

In the Java world three tools are broadly used.

CheckStyles is a static code analysis tool and checks if code is compliant with specified coding rules.

PMD is as well a static code analyzer that looks fro common programming flaws.

PMD is a source code analyzer. It finds common programming flaws like unused variables, empty catch blocks, unnecessary object creation, and so forth. It supports Java, JavaScript, Salesforce.com Apex and Visualforce, PLSQL, Apache Velocity, XML, XSL.

FindBugs scan code looking for bugs.

Potential errors are classified in four ranks: (i) scariest, (ii) scary, (iii) troubling and (iv) of concern. This is a hint to the developer about their possible impact or severity.

Dependency Scan

WhiteSource Bolt is a dependency scanner that provides information on the dependencies that are referenced by a project, like imported vulnerabilities or potential licencing issues.

A nice extension for Azure DevOps makes dependency scanning easy to perform:

In order to test it, a commercial licence is required.

The report highlighted a number of vulnerabilities on my mock project:

Referenced dependencies in my mock application have at least 15 known vulnerabilities.
A detail of the vulnerabilities.

The report also included information about the dependency’s licences:

A breakdown of the licences used by the application.

And a breakdown of the commercial risk:

The risk associated with each type of licence.

Security Scan

OWASP ZAP is an extension that allows performing a dynamic security scan against a running web application.

OWASP ZAP requires a continuous delivery pipeline in place. The extension will pretend to be an attacker and will fiddle the web application with requests to try to identify vulnerabilities.

A great article with a detailed step to step procedure on how to install is available at this link:

I added a separate stage in my CI/CD pipeline:

I managed to run the tool against my mock application with these results:

The security report found different vulnerabilities.

How Much will this cost?

The next question is, how having these four awesome extensions would cost?

Code coverage and security scan may be both be implemented with open source tools as those that have been discussed in this article.

Costs of a dependency scan with WhiteSource Bolt might vary depending on the features that are needed:

WhiteSource Bolt editions.

Prices range from $0 to $28,000.

WhiteSource Bolt prices.

In this article, I tried the Essentials tier.

Regarding SonarQube, I would definitely recommend using at least the developer edition as the community edition can only run on the main branch.

SonarQube editions.

The cost will be proportional to the size of the codebase that has to be scanned:

SonarQube pricing.

The Pipeline I used

For reference I am also using the pipeline that I used to write this article.

The three stages of the pipeline: build, deploy, scan.

Summary

In this article, I looked at how a CI/CD pipeline can have a huge impact on the quality of the code that is written.

Thank you for reading this article, I hope you found it useful.

Everything connected with Tech & Code

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store