Merging Android Unit and Instrumented Code Coverage Reports Offline, using JaCoCo
Code Coverage is an important technique to determine what parts of a codebase are yet to be tested. When testing Android code, you should write tests that do not need an emulator as Unit Tests and those that do require an emulator as Instrumented Tests. One of the benefits of doing it this way is for overall faster test execution (which is great for optimising your Continuous Integration workflows); Instrumented tests are significantly slower to run than equivalent unit tests. However, this means code coverage reports are generated for Unit or Instrumented test suites separately. But what if you want, or need, those reports merged?
This article will:
- Provide a brief primer for what Code Coverage is (intended for newcomers to Code Coverage).
- Motivate why combining Unit and Instrumented (emulator) coverage reports can be useful.
- Demonstrate how to combine Android Unit and Instrumented test coverage reports with a complete working example using JaCoCo in Gradle configurations (for Kotlin and Java).
- Show how it can work in CI/CD workflows and pipelines (with GitHub Actions).
If you are already convinced and yelling “Show me the code!”, skip below to: Combine Android Coverage Reports.
What is Code Coverage?
It is a measure of which lines and branches of code are touched by test code. It is an indicator of how much code is tested, or more appropriately, it is a measure of the absence of testing. Test coverage is a useful tool but does not indicate test quality. Test coverage can be summarised as a percentage, which you may have seen added as a badge to a project’s README.md file:
Coverage reports offer much more detail than a simple percentage value. JaCoCo, for example, generates HTML files for you to browse the source code. The example below shows a JaCoCo HTML coverage report of two source files and how easy it is to read which lines and branches have been tested. The first file has been tested fully and the other has not been tested at all:
If you are new to Code Coverage, then it is interesting to note that there are diminishing benefits the closer you are to achieving 100% test coverage (particularly with large projects). A common approach is to aim for an 80% coverage threshold (certainly consider this a minimum target), but it is wise to aim for quality tests covering as much as possible:
“The 20% you leave uncovered is probably the 20% that needs it the most.”
— Daniel Irvine
Why Combine Coverage Reports?
Separate coverage reports can be preferred for larger projects where you want to analyse those reports for each test strategy separately, but it may add unnecessary complexity to many projects — it depends on the project you are working on and what you want to achieve. Combining code coverage reports can be great for summarising coverage of all test efforts, striking a balance between the benefits of analysing multiple coverage reports independently and reducing the time required for a developer to do so.
Some online services offer to merge coverage reports for you, which is great! However, they usually integrate with CI/CD workflows to upload individual coverage files (so only after a commit to your version control) and then view the results on a webpage after they have been processed. This is not convenient for local or offline development and testing, especially where you may be rapidly iterating through development and testing steps. Who has time for uploading files when you can simply do it on your own computer?
Android Studio has a built-in Coverage feature, but it can only be run for unit tests, not instrumented tests. Even if Android Studio was updated in the future to report both (even combined), those reports are not available in Continuous Integration (CI) environments (as you do not load a heavyweight Android Studio in a CI environment). So we need a lightweight way to combine Android coverage reports in a terminal environment…
Combine Android Coverage Reports
In this sample, coverage reports are generated with JaCoCo and defined as a Gradle Task, so it can be executed in the terminal. This technique builds upon the work of Rafael Toledo (unified coverage report in Android with Jacoco, Robolectric, and Espresso) and Veaceslav Gaidarji (configure JaCoCo for Kotlin & Java), therefore supporting Java, Kotlin, and Java+Kotlin projects.
A complete working example of this demo is available on GitHub:
The following build.gradle snippets should be merged into your existing build.gradle files (not replacing them).
In the project-level build.gradle file, add JaCoCo as a dependency:
In your module-level build.gradle file (e.g., /app/build.gradle), you will configure JaCoCo and declare a Gradle Task that will combine coverage reports when executed:
To execute the Gradle Task, simply run the following in a terminal:
gradlew jacocoCombinedTestReportThe generated HTML (and XML) reports will be located under app/build/reports/coverage/jacocoCombinedTestReports/ (assuming the module name for your application is the default: “app”).
Bonus Level: Merging Coverage in Continuous Integration (CI)
Running instrumented tests in a CI/CD workflow can be tricky if you do not have access to your own private CI runner — software repositories typically offer free CI runners that are configured and/or sandboxed in ways which do not allow emulators to run. This is a complex topic outside the scope of this article, but you can read this explanation and some solutions:
Over the years I have tried a number of claimed workarounds on GitHub and GitLab, but none have ever worked for me…until reading that article (thanks Yang!). I have only confirmed one solution which I will summarise below. If you happen to be using GitHub and are using GitHub actions to run workflows (such as a CI/CD) you can use the following GitHub actions YAML file as a template for running instrumented tests with a hardware-accelerated Android emulator:
This example runs the tests and combines the coverage reports in a CI workflow, using the Gradle task we defined earlier (jacocoCombinedTestReports). You can replace ./gradlew jacocoCombinedTestReports with ./gradlew connectedAndroidTest if you want to just run the plain instrumented tests.
You will still need to upload the coverage reports from the CI runner so they can be used outside of a CI’s execution environment (either for yourself or external services). The demo repository has a complete example of a CI uploading the coverage report to a 3rd-party service (Code Climate, in this example) and uploading the HTML test results and coverage report as a GitHub Workflow Artifact. You can download workflow artifacts after a workflow has finished running.
Alternative services to Code Climate exist, such as the popular Codecov. Regardless of which you choose, you may be required to set an environment variable to be used by your CI (to identify the uploaded files to the service)— I had to set the environment variable CC_TEST_REPORTER_ID. Each service provides instructions for uploading and authenticating with their services (here are Code Climate’s). Alongside their many primary benefits, these services also enable you to easily add a coverage badge to your README.md files, as illustrated in the screenshot of the README.md file at the top of this article. That is a nice little bonus.
