Code-coverage and Carryforward Flags
This is part of a blog series on managing monorepos. If you haven’t already, check out https://medium.com/compass-true-north/repositories-one-or-many-f9da590611af. More to come in the next few weeks!
At Compass, we take the quality of our applications very seriously. We know that it’s key to ensuring a quality user experience for our customers. One way we help ensure our quality is high is to ensure we have proper code coverage for the features we build.
How does code coverage help?
Code coverage is important to us because it is a concrete way to measure how much of our code has applicable tests. While it can’t tell us the quality of these tests, it can show us when critical code paths have no coverage and thus are a risk to break in production due to subsequent code pushes.
A common pushback against tracking code coverage is that just because a line of code gets hit in automated testing, doesn’t mean it will prevent regressions in business logic (e.g., the test could be written poorly but still hit the line). However, maximizing the number of lines hit is the bare minimum you can do to get started, and then build on top of that progress to cover more specific edge cases.
Houston, we have a problem
We use codecov.io to track our code coverage and drill down into areas where our coverage is lacking. However, we had an issue when trying to use the service. We have two large monorepos, but these repositories have numerous, unique applications controlled by individual teams.
When uploading our code coverage, we needed to load the whole repo’s coverage to codecov.io every time we merged to master. The reason for this is that every time we upload to codecov.io, it overwrites the coverage from the previous upload. This loss of data between pushes was a significant problem since generating code coverage for over 70+ applications, and 200+ packages could take several hours and used up tons of our CI pipeline’s resources.
Working around partial test suite runs impacting coverage numbers
We like codecov.io. It’s easy to set up and has a clean UI that makes it easy to drill into the areas where coverage is lacking. However, because of the above issue, it was not working for us. So we reached out to them to explain the problem and what we needed to make it work for us. Codecov.io was very receptive to our feedback. After several discussions, we were able to describe what we needed, and they committed to building a new feature called Carryforward Flags.
Sounds good so far, but what are Carryforward Flags?
Codecov.io provides a YAML file where you define all the rules around how you want to collect the code coverage data and use it. For example, you can specify which code paths you want to include or ignore, if you want it to add pull request comments about coverage, etc. The basic idea with carryforward flags is to have a feature added to the YAML that defines what to upload and when. This feature gives fine-grained control to upload the coverage for an individual application/package. Here’s a simple example in action:
Each flag in the flags section corresponds to a specific application with the monorepo and its relative path. This relationship can become a bit difficult to handle since we have so many apps. As a result, we built a script that auto generates this YAML file and then updates it any time an application is added to or removed from the repo. This functionality creates a pull request that gets reviewed and later merged. We automate as much as possible so that the adoption rate amongst our teams increases quickly.
What were the results?
As a result of the new feature, we can now upload coverage for any application/package. It will not overwrite any of the coverage numbers already in codecov.io for any application(s)/package(s) other than the one your PR makes changes to.
This functionality makes the code coverage tasks in our CI pipeline fast and not resource-intensive. Also, we can now add blocking Github checks for pull requests so that if coverage falls for a specific application, it will not allow the pull request to merge, thus ensuring proper coverage levels. This guardrail is a great way to maintain high test coverage once you’ve paid down the testing debt within your team. For example, adding the following to the YAML file:
Adding the Code Coverage check to a Github PR tells Codecov.io that when changes to the application with the flag app_test are made it should calculate how much code coverage that app now has and if the overall app coverage drops below 75% mark the check as failed (see example below):
So what does this all mean?
With this new visibility, developers and dev managers can now quickly check their progress when it comes to coverage of their application. It allows them to see how bugs/issues that occur correlate to coverage (or the lack thereof). When working in a CI/CD world, moving fast is critical, but it’s also essential when moving quickly to have safety checks. Code coverage that is easy to set up, easy to use, and easy to digest is critical to the stability and high availability of our product offerings and prevents numerous downstream problems for our engineers.
