Combining Java Test Coverage Reports using Gradle
In software development, testing is crucial to ensure that your code is working as intended and to validate that changes to the code base don’t have unintended side effects. However, for huge code bases with numerous classes and test classes, it may be difficult to estimate how much of your code is actually covered by tests. In order to measure the test coverage of your project, tools like Jacoco have been developed. The coverage data can then be exported to other tools, such as Sonarqube to process and aggregate them in an easy to understand and insightful way. In this article we will demonstrate how we use Sonarqube with the Gradle Sonar Plugin in a Java project with Junit 5 tests to gain insights into the test coverage of our project.
Test coverage actually consists of two different coverage metrics; line coverage and branch coverage. The first is defined as the proportion of lines of code that are tested. The latter measures how many branches of your program’s flow are covered by tests. Of course a high coverage does not necessarily mean that your tests are of good quality. A low coverage, however, will let you identify parts of your code which are not tested sufficiently. In order to enforce a certain coverage you can define quality gates in Sonarqube such as a minimum test coverage which should always be fulfilled. For every pull request openened towards your master branch a delta for these metrics is calculated and can be used to enforce that all code added to your project is tested. But how exactly can we achieve that Sonarqube recognizes our tests and their coverage?
In order to display the coverage in Sonarqube we first need to actually calculate it. In fact Sonarqube does not actually measure the coverage but merely relies on the code coverage calculated by Jacoco (short for Java Code Coverage). Jacoco is a library that attaches itself to the JVM that runs your tests. It operates on the Java class files which are essentially byte code instructions. Jacoco establishes different counters to track which class files are used by the execution of your unit tests. These are then mapped to the original code and Jacoco provides you a report to show how extensively the different parts of your application are tested.
Calculating Code Coverage with Jacoco
Let’s have a look at our Gradle test task and how we configure jacoco to measure code coverage.
Our Gradle file consists of two different tasks, test and jacocoTestReport.
In the test task we first declare that we use Junit 5. Then we add a Jacoco configuration block. In order to produce a coverage report that can later on be fetched by Sonarqube we define the destinationFile and the classDumpDir. destinationFile specifies where the exec files which contain the measured coverage are stored. As you can see in the jacocoTestReport task we instruct Jacoco to look for execution data stored in the same directory we have just specified as destinationFile. classDumpDir is actually not relevant for Sonarqube but makes debugging the coverage generation easier. For more info about Jacoco params please have a look at the Jacoco Gradle Plugin guide. You can also check this page which lists and explains the different parameters for the Maven plugin but AFAIK these explanations are valid for the Gradle plugin’s configuration parameters, too.
Next, we configure the jacocoTestReport task. Jacoco offers different representation of its results, e.g. a xml and a csv report. Sonarqube will only work with the xml report, so this has to be enabled. You can enable the other reports as well, i.e. for quick usage from your local machine but they do not affect Sonarqube in any way.
When we execute our unit tests, Jacoco will produce a jacocoTest.exec file which contains a coverage measurement for our application code. Furthermore, it will generate jacocoTestReport.xml in the directory build/reports/jacoco/test. Now we need to import this report into Sonarqube in order to get a visual representation, to employ quality gates and to get a history of how the coverage has changed compared to prior builds.
Integrating the Gradle Sonarqube plugin
In order to include the Sonarqube plugin into our Gradle project we add the following line to our plugins block.
Then we need to configure the sonarqube task by setting several Sonarqube properties.
Let’s shortly describe what these properties mean and why they are important.
- sonar.host.url: Here you can define the address of your Sonarqube instance.
- sonar.projectName: The name of the sonar project. By default this is your Gradle project name
- sonar.projectKey: The key of the sonar project.
- sonar.branch.name: Here you can a define a branch name. This allows you to compare the Sonar analyses between two different branches. The example code uses a placeholder which is replaced with your branch name automatically. Note that this is not available in the Sonarqube community edition.
- sonar.sources: Tells Sonarqube the path to your source code.
- sonar.coverage.exclusions: Specify classes which should not be included in the coverage calculation.
Additionally, there are two more properties which are defined automatically when both, the Sonarqube and the Jacoco plugin, are applied to a project in combination. Both, sonar.jacoco.reportPaths and sonar.groovy.jacoco.reportPath are per default set to the Jacoco destination file path jacoco.destinationFile. As we remember, jacoco.destinationFile is the property which we have set in the Jacoco configuration in tests.gradle. Since Sonarqube is always listening to the Jacoco path and we set the Jacoco explicitly there is no need to configure anything Jacoco specific in our Sonarqube configuration. Instead we can simply execute the sonarqube task the Sonarqube plugin provides and will see the coverage of our unit tests.
So far so good. Let’s recap shortly what we have achieved so far:
- We have setup a basic test task with Jacoco support such that coverage metrics are calculated for unit tests.
- Then we have setup a Sonarqube configuration to let Sonarqube analyse the coverage metrics provided by Jacoco
However, there are not only unit tests but also a bunch of other test categories. Although some of these tests may be external tests that are not part of your CI pipeline, especially integration (and potentially contract tests) should definitely be part of every of your builds. But how should we assess how well your system is tested without considering their code coverage as well? Thus we will show how we can instruct Jacoco to measure coverage for these tests as well.
Although this task does not set any Jacoco specific properties we will see that integration tests are now part of your coverage calculation. This is caused by the inheritance of the test task type. Every Jacoco properties set in the original test task will be set for every task that extends it. When we execute both tasks, test and integration, Jacoco will produce two .exec files, jacocoTest.exec and integration.exec. All we need to do in order to allow Sonarqube to fetch both files is to alter the following line in our tests.gradle:
getExecutionData().setFrom(fileTree(buildDir).include("/jacoco/jacocoTest.exec"))by replacing jacocoTest.exec with a wildcard *.exec.
getExecutionData().setFrom(fileTree(buildDir).include("/jacoco/*.exec"))Then Sonar will scan for all .exec files and calculates a combined coverage for all of them.
Conclusion
In this article we have shown how we can get aggregated test coverage for different types of tests. First we have introduced all required tools. Jacoco, to collect the coverage data, and Sonarqube, to display and aggregate the different test classes’ coverage information. Subsequently we have shown how we can integrate the Sonarqube plugin into a Gradle build file to calculate the coverage automatically for each task that extends the Gradle test task.
