Measuring unit test coverage in multi-module Android projects using Jacoco: Part 2

In the first article, we discovered one of the two key Gradle commands which come with Jacoco plugin — jacocoTestReport. As you now know, it can be used to measure code coverage in your project by generating detailed reports in HTML, XML, or CVS formats. This time I’d like to share my experience with the second piece of this puzzle — jacocoTestCoverageVerification.

As per the documentation, this task does the following:

Task for verifying code coverage metrics. Fails the task if violations are detected based on specified rules.

So as you might have already guessed, this tool can be used to check whether some coverage threshold is reached or not. Basically, you specify the minimum coverage measured in % and run the task. If your unit tests “visit” the desired % of your codebase then this task will silently complete. Otherwise, it will fail. This can be really helpful for development teams since it allows you to have a kind of enforcement for writing unit tests and all you need to do for that is to setup Jacoco and then simply ask your CI/CD to execute jacocoTestCoverageVerification along with other tasks for your pipelines.

Let’s get started with the configuration. I’ll continue from where we stopped in the previous article. As with jacocoTestReport, it would be still useful to have two different setups for Java/Kotlin and Android modules. Let’s add two more functions to the jacoco.gradle file.

def setupAndroidCoverageVerification(threshold) {
    task jacocoTestCoverageVerification(
            type: JacocoCoverageVerification,
            dependsOn: [ 'testDebugUnitTest' ]
    ) {
        violationRules {
            rule {
                limit {
                    minimum = threshold
                }
            }
        }
        final def coverageSourceDirs = [
                "$projectDir/src/main/java",
                "$projectDir/src/main/kotlin"
        ]
        final def debugTree = fileTree(
                dir: "$buildDir/tmp/kotlin-classes/debug",
                excludes: jacocoFileFilter
        )
        sourceDirectories.from = files(coverageSourceDirs)
        classDirectories.from = files([debugTree])
        executionData.from = fileTree(
                dir: project.buildDir,
                includes: ['jacoco/testDebugUnitTest.exec']
        )
    }
}
def setupKotlinCoverageVerification(threshold) {
    jacocoTestCoverageVerification {
        dependsOn test
        violationRules {
            rule {
                limit {
                    minimum = threshold
                }
            }
        }
    }
}

Each of these functions registers jacocoTestCoverageVerification Gradle task for a given module. This task is derived from the JacocoCoverageVerification type and depends on testDebugUnitTest which means that launching the verification task will also execute your unit tests before doing the validation (which is quite expectable: we need to run all the tests before we can measure their coverage). These functions also have the threshold parameter so we can dynamically provide it from the calling side.

It is time to remember that we’re talking about the multi-module Gradle setup so hardcoding a single coverage threshold may not be a very good idea: most likely many modules would want to customize this value on their own based on the module type, stage of development, etc. The simplest way to achieve this is to define constants with a fixed name in module-level build.gradle files and then read its value during Jacoco configuration:

Module A’s build.gradle:

...
ext {
    jacocoCoverageThreshold = 0.6 // 60%
}

Module B’s build.gradle:

...
ext {
    jacocoCoverageThreshold = 0.8 // 80%
}

And back to our jacoco.gradle:

afterEvaluate { project ->
    def ignoreList = jacocoIgnoreList
    def projectName = project.name
    if (ignoreList.contains(projectName)) {
        println "Jacoco: ignoring project ${projectName}"
        return false
    }
    def threshold = project.hasProperty('jacocoCoverageThreshold')
            ? project.jacocoCoverageThreshold
            : project.jacocoCoverageThresholdDefault
    if (isAndroidModule(project)) {
        setupAndroidReporting()
        setupAndroidCoverageVerification(threshold)
    } else {
        setupKotlinReporting()
        setupKotlinCoverageVerification(threshold)
    }
}

Note that jacocoTestCoverageThresholdDefault value will be used if some modules don’t want to provide a custom threshold and are fine with the project-level defaults. Where this default value can be stored? Let’s power-up thejacoco-ignore-list.gradle from the previous article by converting it into a more global jacoco-config.gradle:

project.ext {
    jacocoCoverageThresholdDefault = 0.60
    
    jacocoIgnoreList = [
            "module-name-1",
            "module-name-2"
    ]
    // Exclude file by names, packages or types. Such files will be ignored during test coverage
    // calculation
    jacocoFileFilter = [
            '**/*App.*',
            '**/*Application.*',
            '**/*Activity.*',
            '**/*Fragment.*',
            '**/*View.*',
            '**/*ViewGroup.*',
            '**/*JsonAdapter.*',
            '**/di/**',
            '**/*Dagger.*'
    ]
}

We’re done with the basic setup: all modules now support jacocoTestCoverageVerification task. As with jacocoTestReport, it can be launched for individual modules by executing the following command

./gradlew module-name:jacocoTestCoverageVerification

Or you can verify all the modules at once by typing

./gradlew jacocoTestCoverageVerification

Now you can quickly find modules with no tests and blame their authors, right? Not yet, but we’re close! :) It appeared that Jacoco and both its tasks ignore modules that don’t have any tests added, i.e. jacocoTestCoverageVerification will not fail with a “0% test coverage” error for such modules for some reason, it will only fail if this module contains at least 1 unit test and this test doesn’t provide the desired code coverage. Let’s add a custom workaround for this limitation.

First, create a new task inside jacoco.gradle :

class TestExistenceValidation extends DefaultTask {
    static final SRC_DIR = 'src'
    static final JAVA_DIR = 'java'
    static final TEST_DIRS = ['test', 'androidTest']
    
@TaskAction
    void execute() {
        if (shouldSkip(project)) return // implement `shouldSkip` as required for your project or just remove this line
        File srcDir = new File(project.projectDir, SRC_DIR)
        FileFilter filter = { it.isDirectory() }
        File[] subDirs = srcDir.listFiles(filter) ?: []
        File testsDir = subDirs.find { TEST_DIRS.contains(it.name) }
        if (testsDir) {
            File javaTestsDir = testsDir
                    .listFiles(filter)
                    .find { it.name == JAVA_DIR }
            if (javaTestsDir && javaTestsDir.list().length > 0) {
                return
            }
        }
        throw new GradleException("${project.name} has no unit tests.")
    }
}

This task will simply make sure that each module contains non-empty test folders (either test or androdiTes). This logic can be further extended if needed, but I decided to don’t be too paranoid about that. Now we need to link this newly created testExistenceValidation to the verification task. As you already know, it can be achieved by usingdependsOn:

def setupAndroidCoverageVerification(threshold) {
...
   dependsOn: [
        'testExistenceValidation', // repeat for setupKotlinCoverageVerification
        'testDebugUnitTest'
   ] 
}

Done! Our verification task now also depends on testExistenceValidation routine and it will automatically execute it before doing its main job.

P.S.: complete versions of all source files I mentioned here can be found in this Gist.