Multi-Language, Single-Repository Code Coverage Analysis, Made Simple*!

*if you are using Travis CI and have access to Ruby. 😁

Code coverage isn’t very sexy, but it’s a good-enough indicator of the likelihood that major bugs will creep into a codebase. The closer you get to 100% — so long as you also are writing thorough, non-BS tests — the less likely that a showstopper issue will rear its head and ruin everyone’s day.

A task came across my plate to figure out a strategy for bundling and shipping our code coverage reports from our various subcomponents (stored in a monorepo) into Coveralls. Sounds simple, but most solutions assume a single language is being used for an entire repository; the advent of microservice architecture makes that reality less likely.

Step 1: Identify the common denominator

There are a dizzying array of coverage output formats out in the world: lcov, gcov, HTML, Simplecov… the list goes on. After much research, I was able to identify a particular format that most languages have an output adapter for: Cobertura XML. In my case, I was juggling Python and JS code which use coverage.py and Jest (by way of Istanbul) respectively to generate the appropriate reports — each has an xml emitter that happens to match up to Cobertura specifications.

With a common format in mind, the next step was to write a script to carefully merge the output.

Step 2: Getting friendly with XML

The official Coveralls integration is Ruby-based, so I decided not to rock the boat and write a small Ruby script to aggregate the Cobertura results into the final payload the Coveralls JSON API desires.

Thankfully, Ruby has Nokogiri (a dead simple XML parser.)

(Part 1 of 2) The locations of each generated cobertura XML are provided to the Ruby script, which analyzes them, merges them, and fetches the original source files to do per-line hit statistics that Coveralls needs.

Step 3: Ship it, wash hands, never look at XML again

Now that we have the assembled payload, all that’s left to do is use the official Coveralls gem to ship the output. My team uses Travis CI for testing automation, so all that needs to be done in that case is call the appropriate gem method and all is well.

(Part 2 of 2) With the array built by the previous script fragment, we use the official Coveralls gem to send the assembled payload.

If you are not using Travis, more setup might be needed.

Here is the final version of the script:

Note: The script requires the coveralls and nokogiri Ruby gems to be installed.

TL;DR I made a Ruby script that cobbles together Cobertura XML from different parts of a codebase, assembles a Coveralls-compatible JSON payload, and sends it off using the coveralls gem. Here it is: https://gist.github.com/probablyup/9ff110729ea73add55c887623708fd90