Increasing test coverage without slowing down development

Releasing more frequently requires investment in test coverage, but you don’t have to stop shipping features to achieve this

TLDR

The short answer:

• Set a Quality Gate for Test Coverage in CI/CD pipeline
• Fine tune threshold value according to the project state to match current coverage (this is usually a low value depending on the amount of technical debt)
• Gradually raise the threshold to the appropriate level

For the truly impatient, check steps here: How to set up a Test Coverage threshold in Go and Github

Why is this needed?

Application Modernization initiatives and DevOps culture are forcing app teams to embrace frequent changes and continuous delivery. In order to preserve a good level of quality and confidence while refactoring source code, a fast and robust feedback mechanism is critical so regression issues can be addressed right away. Test Driven Development (TDD) combined with CI/CD pipelines is a great approach. This is usually translated into writing a lot of unit tests and aiming for high test coverage, then running tests automatically as part of continuous integration and halting the process in case of issues.

While achieving high test coverage is desired, this is usually an incremental process and the actual target varies from team to team. This also depends on type of project: green field (brand new code) vs brownfield (modernization of existing/legacy code). Green field projects have the opportunity to adopt a TDD approach, in which the evolution of code should go hand in hand with an appropriate number of tests. Brownfield projects, on the other hand, often deal with technical debt and teams must deliberately dedicate resources to write tests for existing code, which in some cases means stopping work on features.

A flexible yet consistent strategy is needed to providing a path to a good target state in terms of test coverage, while avoiding too much disruption for the team and product.

Fixing the “water leak”

The Sonarqube team coined this scenario as Fixing the water leak (recommended article). In a nutshell, the idea is to ensure that current quality level is preserved for legacy code, while providing good test coverage for all “new code”. This way, the quality never gets worst than it is now, and should only get better overtime.

Introducing Quality Gates

A Quality Gate is a mechanism by which a team sets a threshold for a particular quality metric (i.e: number of defects, cyclomatic complexity, test coverage, etc). If actual value goes below threshold then delivery stops until the quality attribute is back on track. Quality Gates can be embedded into CI/CD pipelines to automatically trigger correction mechanisms.

For this particular scenario, setting a quality gate for Test Coverage is part of the solution. The other part is how that metric is handled effectively.

Setting the right metric

Enforcing a high arbitrary value for test coverage would cause team members to stop working on features and dedicate all the time to unit testing, which could also lead to poorly written tests (if the goal turns to simply hit “that metric” then the metric will cease to be effective). Instead, the suggestion is to set a lower “threshold” based on current state, which would allow CI/CD pipeline to pass, but would fail if coverage goes below it. This approach ensures that overall “code quality” never gets worst than it is now. The threshold can be reviewed periodically (i.e.: during retrospective meetings) and gradually increased when the team feels is appropriate.

Sample implementation

• How to setup Test Coverage threshold in Golang and Github
  This post contains specific steps to setup a Test Coverage Quality Gate for a particular scenario: Golang code base with Github Actions as CICD pipeline.

Summary

Development teams often face the challenge of moving at a fast paced while ensuring quality and stability. As described above, a Quality Gate is a very useful technique for improving code base over time without stopping progress on new features. This is more applicable to “brownfield projects” dealing with legacy code and infrastructure, since brand new projects can adopt a TDD strategy from the start. While the implementation described here referred to a very specific scenario, the approach is generic and can be applied to any project and code base. Automated tools such as SonarQube are readily available and cover a wide range of coding languages.

Are you confident about refactoring your legacy project? What are you doing to ensure stability without sacrificing speed?