Targeted Code Coverage Reports for Android Continuous Builds
By: Brian Terczynski
Automated testing is an important part of software development. Developers should always write tests for the code they produce. This not only ensures that their code is correct, but will also help catch any future regressions as the codebase evolves.
Unfortunately, developers do not always do a good job of this. Often, they will only write tests for simple, happy-path cases. They will not cover edge cases, error cases or even different branches in the code. There are even times when developers do not write tests at all.
Code Coverage Reports
One way to catch insufficient testing is with code coverage reports. Code coverage reports indicate which instructions were executed when you run your tests. In particular, you can use them to know which instructions were not executed, which can give you insight into which test cases should be added to properly test your code.
The Android team at Thumbtack gets its coverage reports via two means: (1) from within Android Studio, and (2) from our continuous integration (CI) build.
The code coverage reports in Android Studio allow us to run individual unit test classes and see what the coverage is for the code-under-test. The CI reports show the total coverage for the entire codebase for a particular change request (CR) based off some parent commit.
What is not obvious from those two mechanisms are: (1) what is the coverage for just the particular lines of code that were changed (which we will call the CR-specific coverage), and (2) what is the delta in code coverage between this particular CR and the parent commit (which we will call the overall coverage delta). CR-specific coverage is useful because it narrows down just the particular lines in the CR that lack sufficient coverage, versus showing coverage for the entire codebase (which can be quite large). The overall coverage delta is useful as it lets developers know if they are regressing code coverage for the entire codebase with their CR, and which lines are the cause of that change. Both of these reports are specific as to which lines of code need tests, versus trying to glean that information from a monolithic coverage report.
Really, what we want are succinct reports of which lines of code need more coverage. We want developers to easily know which tests they should add for their CR. To do that, the Android team at Thumbtack added both a CR-specific coverage report and an overall coverage delta report as part of their CI pipeline.
CR-Specific Coverage Report
When a CR is put up for review, automated tests are run on that CR. If those tests pass, we generate an overall coverage report. We then get the intersection of that report with the lines of code that were changed in that CR , to determine which of those lines in that CR do not have code coverage. This constitutes the CR-specific coverage report. This allows the author and reviewers to know specifically, and succinctly, which lines need more coverage. That way, the author and reviewers can more easily determine what additional tests should be added.
Coverage Delta Report
In addition to that, after the CR’s tests are run and coverage is generated, we then get the coverage for the parent commit, and do a diff to show which files have changed coverage, as well as the overall change in coverage. This is our coverage delta report. This information lets the author and reviewers know if the CR is regressing coverage in our codebase, and where the author might want to add tests to prevent this.
Implementation
Both of these reports are generated via Python scripts that we wrote in-house. The reason we opted to write our own, and not go with a third-party tool, was because the scripts themselves were simple enough to develop (each around 150 lines of code), and we could write them to work well with our existing CI system and code-review tools (for which third-party tools did not integrate easily or were too costly). We already use Jacoco as our industry-standard coverage tool for Kotin/Java code. Our scripts simply integrate with that tool and Git to generate our targeted reports. For example, the coverage delta report involves simple parsing and comparison of the Jacoco XML files for each build. The CR-specific coverage script maps the lines changed in the CR to the line coverage data in the Jacoco XML file. Each script outputs the data in a simple HTML format for our consumption, as well as custom messages sent to our code review tool.
Results
Since adding these reports, we have started to see an increase in the number of unit tests added to CRs. With the CR-specific coverage report, it has become more obvious to authors and code reviewers where tests have been missing. We have seen authors amend their CRs with additional tests after seeing these reports, and reviewers have been able to give more specific feedback on tests that should be added. The coverage delta reports have also been useful for that purpose, but have also helped when we would retroactively add missing unit tests. They were particularly useful during our recent “Android Test-Writing Days”, where a few engineers spent time adding missing unit tests at a rate of >1% coverage increase per day. And finally, since these reports are recorded in our CR history, we have been able to go back to these reports to see how we have done over time.
Conclusion
Succinct coverage reports provide useful and more consumable information to see how CRs impact code coverage. When integrated into your CI pipeline and code review tools, they can provide direct advice on additional tests that should be added. As there are many different CI and SCM systems out there, consider if a third-party tool for coverage comparison will work for your case. Or you can write simple scripts to compute that data, like we did at Thumbtack, as you can then customize them to work with your existing infrastructure and tailor them as you see fit. Whatever approach you choose, targeted coverage reports can definitely help both code authors and reviewers get a better idea of where testing coverage can be improved.
Originally published at https://engineering.thumbtack.com on December 2, 2019.
