Automated Code Coverage & Quality Gate Analysis for Maven Multi-Module Project on Bitbucket using JaCoCo, Sonar & Jenkins
Analyzing code coverage of a maven multi-module project using JaCoCo while making use of Jenkins and Sonarcloud to automate the analysis of Quality Gates in PRs in bitbucket repositories.
Introduction
This blog will help you understand the following for a mono or multi-module maven project -
- Code Coverage using JaCoCo
- Publishing analysis to Sonar (using JaCoCo XML Report)
- Quality Gate Analysis on Sonar
- An Exceptional Case where test cases covering code for one module are written in a different module.
- Automating the analysis of PRs on Bitbucket Cloud using Jenkins
We have created a GitHub repository with all the required dependencies to generate the JaCoCo Report, you can visit it here. The code will help you understand this blog a little better.
We also use the latest version of plugins available at the time of writing this blog post to ensure that the code is standardized, efficient, and neat.
At the time of writing this blog, the JaCoCo maven plugin was at 0.8.7 and Sonarqube at 8.8.
What is code coverage?
Code Coverage is a metric used in software engineering to describe the number of lines of code that have been covered by a particular testing suite. It also checks whether all branches of the code have been covered. A higher code coverage, measured as a percentage wrt total lines of code, has had most of its source code executed during testing which suggests that it has lower chances of containing bugs as compared to one with lower test coverage.
JaCoCo Maven Plugin
JaCoCo is an open-source toolkit for measuring and reporting Java code coverage. JaCoCo is distributed under the terms of the Eclipse Public License. The JaCoCo Maven plug-in provides the JaCoCo runtime agent to your tests and allows basic report creation.
In our case, we have a maven project with java and some unit tests written in JUnit or TestNG. JaCoCo will help us analyze which of the lines of code are correctly tested and which aren’t. This is where the JaCoCo Maven Plugin comes into the picture. It offers many different goals, you can read more about them here. For our use case, we are only concerned with the below -
- jacoco:prepare-agent — Prepares a property pointing to the JaCoCo runtime agent that can be passed as a VM argument to the application under test.
- jacoco:report — Creates a code coverage report for tests of a single project in multiple formats (HTML, XML, and CSV). This report (XML) is also used by sonar to analyze the coverage.
- jacoco:check — Check that the code coverage metrics are being met.
We will be going into more detail about these later.
Dependency
To automatically run the above JaCoCo goals we will add the plugin to the build part of the pom.xml of our maven project. If you have a multi-module project, ensure that you add this to the parent pom.xml.
After the addition of this plugin, whenever you run mvn clean verify the JaCoCo report will be generated inside the target/site/jacoco directory. Inside this directory, you will find an index.html and a jacoco.xml file. The HTML file can be used to view your code coverage as shown below whereas the XML file is mainly used by Sonar to analyze the coverage.
The report generated contains details about every line of code as well as branches that has or hasn’t been covered -
Some more information on the goals used to generate this report -
prepare-agent — As mentioned before, this goal prepares a property pointing to the JaCoCo runtime agent that can be passed as a VM argument to the application under test. One of the ways to do this in the case of maven-surefire-plugin is to use syntax for late property evaluation:
Then this argLine is provided to JaCoCo maven plugins’ prepare-agent phase -
This goal generates the jacoco.exec file which stores the execution data of the test cases that ran.
report — As mentioned above this phase generates the report in XML and HTML formats. This phase requires the jacoco.exec file which was generated in the prepare-agent phase. There are various optional parameters that can be provided in this phase which can be found here.
check — This is another goal that is optional. It can be used to fail maven builds in case some provided rules fail.
More compatible rules can be found here.
In case you have a multi-module project after you run mvn clean verify for the parent module, you’ll be able to find target/site/jacoco in each module which should contain the coverage for code in that particular module, also the coverage check will run for each module individually.
Sonar Analysis
To perform sonar analysis you will either need Sonarqube or Sonarcloud. You can set up a server of Sonarqube very easily using their official documentation. For more details on Sonarcloud, you can visit https://sonarcloud.io/.
SonarSource Scanner Maven Plugin
SonarSource provides a maven plugin to help scan and analyze our code, including coverage. Note: This plugin requires JDK11.
You can find the latest version here.
After adding the plugin, we also need to add some default properties about sonar to the parent pom.xml as follows -
Now, we need to execute the following command from the root of our project to have sonar scan it:
mvn sonar:sonar -Dsonar.login=<sonar-token>In case you want to exclude any files from sonar coverage analysis or sonar analysis altogether, you can add the following properties -
<sonar.exclusions>**/*.xml</sonar.exclusions><sonar.coverage.exclusions>**/dto/**/*</sonar.coverage.exclusions>Analysis Result
Once the analysis completes, you can open your project in your sonar server which should look like the following -
This page will provide a complete report on your latest analysis, inclusive of bugs, tech debt, security vulnerabilities, code coverage, etc. You can view issues as well as coverage for all classes and lines of code using the provided tabs.
Sonar Quality Gates
What are quality gates?
Quality Gates are basically a set of conditions defined on a project which need to be passed for the quality gate to be marked as passed. These conditions ensure that the “new” code is of production quality, that is, it doesn’t introduce new tech debt, security vulnerabilities, etc by analyzing the new code and also ensuring that the new code has been covered by test cases.
By default, sonar provides the sonar way quality gate which has the following conditions -
Defining Custom Quality Gates
To define a custom quality gate you can click on the “Quality Gates” Tab and then click on “create” to define a new quality gate. You can now add a variety of conditions provided by sonar that can be included as part of this quality gate. You can also use the projects subsection to define which projects will be making use of this quality gate.
Exceptional Case
What if I have dependencies among my sub-modules and test cases for a dependent module are written in the module that imports it?
In our case we had a module, let’s call it “module-a”, and another module called “module-b”. module-b has a dependency of module-a and test cases covering code for both these modules are written under module-b. In this case, if we use the above approach the jacoco analysis for module-a will not generate any report as it does not contain any test cases and the analysis for module-b will generate a report only for the code present in the module-b and not module-a.
In such a case we need to first tell module-b to unpack dependency (classes) of module-a. For this, we make use of the maven-dependency-plugin. We will add this module to the pom of module-b and not the parent pom.
Once we have added this plugin, when you run mvn clean verify again, you will notice that the target/site/jacoco folder of module-b now also has the packages of module-a because of which the index.html (report) generated now shows coverage for packages and classes of module-a and module-b.
Now, we also need to tell the SonarSource plugin that when it’s analyzing module-a, it should refer to the JaCoCo XML report generated under module-a and not module-b. For this, we need to add the following property to the pom.xml file of module-a.
<sonar.coverage.jacoco.xmlReportPaths>
${project.basedir}/../module-b/target/site/jacoco/jacoco.xml
</sonar.coverage.jacoco.xmlReportPaths>Now when SonarSource plugin analyses module-a, it will take the coverage report for the relevant classes from the target folder of module-b which contains the coverage of the classes of module-a being analyzed.
Analyzing Bitbucket PRs using Jenkins
In our case, we use Bitbucket webhooks and Jenkins multi-branch pipeline in addition to the Jenkins sonar plugin to run 0-click sonar analysis and generate quality gates on all our PRs. This ensures that PRs aren’t merged if the Quality gates fail.
You can use the documentation provided here to learn how to trigger multibranch pipelines in Jenkins from bitbucket.
Following is the Jenkinsfile which runs whenever some code is pushed to a branch of the respective bitbucket repository -
Note: In our Jenkinsfile, we have a line where we fetch the latest target branch from git, this is needed because while analyzing Sonar needs the latest code in the target branch to compare and analyze the new code from the current branch to the target branch.
Below is how our PR looks in bitbucket -
In our repository setting, we have defined to not allow merges if any build fails and mandated to have at least one build as passed.
Since we use Sonarcloud, our bitbucket projects are linked to the respective project in Sonarcloud under our organization. This allows for sonar to leave some comments on our PRs as well, below is how it looks -
You can refer to https://sonarcloud.io/documentation/getting-started/bitbucket-cloud/ to get more information on how the bitbucket cloud and Sonarcloud work together.
Conclusion
Using the latest versions of the JaCoCo maven and SonarSource maven plugins really made the code coverage generation and analysis relatively cleaner, neater, and more understandable.
The use of Quality Gates in our PRs also made code reviews simpler and we no longer needed to review for any tech debt, bugs, etc, and could check for more important things like design and logic. It also helped ensure that no security vulnerabilities are being added to the code.
The comments added on PRs by sonar also make it easy for the developer to resolve them quickly as well as understand why that is an issue.
