Generating Android/JVM aggregated coverage reports

April/2023 update
The content of this article is now available as two Gradle plguins:
- io.github.gmazzo.test.aggregation.coverage for jacoco-report-aggregation support
- io.github.gmazzo.test.aggregation.results for test-report-aggregation support

A modern tooling build script filling the gaps between Gradle and Android Gradle Plugin (AGP) to provide a unified coverage report.

The Gradle team recently introduced the JaCoCo Report Aggregation Plugin, which works upon the JVM Test Suite Plugin (default applied by the Java Plugin) to aggregate coverage data produced by a multi-module build into a single coverage report. It only support the JVM modules use case.

As usual, the AGP team is a bit behind on Gradle’s latest features. They recently introduced native support for code coverage reports adding a createVariantUnitTestCoverageReport task per buildType having enableUnitTestCoverage = true .

Even AGP now has the capability of producing coverage reports natively, they are not (yet?) integrating with Gradle’s official aggregation mechanism for this (which is producing outgoing variants, we’ll cover it later).

Before going into technical details, let’s go straight to the solution to our problem.

Adopting the Coverage plugin

On my demo repo, you can find a copy of coverage.gradle.kts buildSrc ad-hoc plugin, that you can easily adopt on your projects to support a hybrid Android/JVM modules aggregated report.

Just copy coverage.gradle.kts (and its companion DSL CoveragePluginDSL.kt) into your buildSrc/src/main/kotlin (make sure to apply kotlin-dsl plugin) and then apply the id('coverage') plugin at your root’s build.gradle

A root :jacocoTestReport task will be registered, aggregating by default the test coverage of all Android modules and all JVM modules (that applies jacoco plugin)

A quick cheatsheet for adopting the plugin:

1. Apply the coverage plugin at the root project
2. Every Android module debug variant will be automatically aggregated (while testCodeCoverageReport is kept to its default true )
3. Every JVM module that also applies the jacoco plugin will be automatically aggregated

For a complete working example, check out the demo project here:
https://github.com/gmazzo/android-jacoco-aggregated-demo

What does this plugin do?

Basically, it fills the gaps between JaCoCo Report Aggregation Plugin and AGP, by providing the missing outgoing variants for any variant that has native coverage reports enabled.

To understand better what this plugin does, let’s first focus on how JaCoCo Report Aggregation Plugin on JVM modules:

When jacoco-report-aggregation is applied, a jacocoAggregation configuration is created. If this is done in a root project (that does not apply a java plugin), you need to manually configure the target aggregated report to generate and from which modules will it feed.

For instance, by adding dependencies { jacocoAggregation(project(“:child”)) } you’ll tell the plugin to include the project child when generating the aggregated report. You can do this for every project (potentially all of them) in the build.

Later, the task associated with the report (matching the report name, testCodeCoverageReport if you following Gradle’s official example) will resolve jacocoAggregation fetching a special outgoing variant for each of its dependencies, that matches the coverage report attributes:

You an inspect a module’s outgoing variant by running the task ./gradlew mymodule:outgoingVariants

What about Android modules?

By default on AGP logic, turning enableUnitTestCoverage on will create the required Gradle tasks to generate a coverage report. But not the required outgoing variants.

Even if you add your Android module as a dependency of jacocoAggregation, the JaCoCo Report Aggregation Plugin won’t be able to resolve it to fetch the JaCoCo execution data file (the build/jacoco/test.execartifact of the coverageDataElementsForTest outgoing variant for JVM; on Android it actually will be build/jacoco/testVariantUnitTest.exec if enableUnitTestCoverage is enabled)

That’s what the main gap this plugin is meant to fill: to provide the required outgoing variants for each Android module, allowing them to get discovered by the JaCoCo Report Aggregation Plugin.

In concrete, the plugin is adding 3 outgoing variants on each Android module:

codeCoverageExecutionData is the main entry point and basically, it tells where to find the JaCoCo execution data file.

JaCoCo also requires two additional inputs to compute a module: sources and classes. The plugin also provides the codeCoverageSources and codeCoverageElements variants to supply them:

The Android’s multi-artifact problem

One thing that is not trivial in comparing regular JVM modules vs Android ones, is the concept of variants.

JVM modules are easy, they just produce a single JAR. Android, on the other hand, the bare minimum will produce two main artifacts: debug and release.

JaCoCo is the default tool used by Gradle and Android to produce coverage reports. One of the limitations this tool has when creating reports is that it will fail if two classes with the same canonical name are found on different .exec files:
java.lang.IllegalStateException: Can't add different class with same name: org/hamcrest/BaseDescription

How this is a problem for Android? well any class under src/main will be present on both debug and release artifacts (assuming both have enableUnitTestCoverage enabled). The problem goes deeper if you have two different versions of the same class under src/debug and src/release respectively.

The is no correct workaround for this but just keep 1 copy of the divergent classes for the final aggregated report. The aggregated report won’t be fully accurate, but still, it will be a good approximation.

The plugin overcomes this by introducing an allVariantsClassesForCoverageReport intermediate Copy task, feeding from all aggregating variants having coverage reports enabled. This task has duplicatesStrategy = DuplicatesStrategy.WARN to let the consumer know about this potential inaccuracy, but still allowing the report to succeed.

By default, the plugin will only enableUnitTestCoverage on debug. Keep in mind that enabling it for more than one variant will make all tests under src/main to run more than once (once per variant’s Test task)

Flavors makes the configuration more complex

As you know, variants are the cartesian product of buildTypes and productFlavors.

enableUnitTestCoverage is a buildType setting, meaning all variants of it will produce a build/jacoco/testVariantUnitTest.exec file.

Even the plugin will support aggregating any number of variants in the root report, I strongly recommend only choosing one of them, to produce consistent reports.
You can use the dedicated aggregateTestCoverage DSL (available on buildTypes and productFlavors) to make that choice.

Look at the documentation for further reference on how to use it.

Are you a SonarQube user?

Thanks that now you have a root :jacocoTestReport that considers alls modules, you can easily report to it by adding the following build logic:

Why it’s not released at Gradle Plugins Portal?

Well, because I saw in the last year AGP team catching up, like introducing native coverage reports or even test fixtures. I think it’s a matter of time for them to include this missing piece as part of AGP logic, making a possible plugin obsolete.