Research on code coverage

Answers to three important questions of code coverage and important industry data points.
Join the discussion on reddit. Make sure to check our ebook: the ultimate guide to code reviews.
We also just released a security dashboard for your team in Codacy.

A nice coverage of red seats.

Best practices dictate that you should track code coverage. Code coverage helps you assess the proportion of code that was tested and (more importantly) not tested.

Why is it important to track coverage? Our users report that coverage is important because it is the confidence level of code being correct and with less bugs. We’ve written previously about code coverage, this is a follow up based on industry data.

In this article we share some evidence of how different companies do code coverage, based on a survey of 680 companies about their code quality practices. We’ll answer three questions:

1. Who tracks code coverage? What happens when you don’t?
2. What thresholds are used for code coverage?
3. What workflow rules are implemented with code coverage?

Without further ado…

Who tracks code coverage? What happens when you don’t?

Let’s first look at some of the numbers from our research.

Code coverage breakdown

Half of the companies track code coverage. We’ll see in a second how that impacts code quality and other parts of development.

This means that half of the companies cannot trust their code to be without bugs.

The bigger the company the more likely it is to track code coverage, as shown in the chart below.

Coverage tracking by company size

You can see that the larger the company the more likely it is to be tracking code coverage.

Not surprisingly, the older the company the more likely it is to track code coverage too.

Coverage tracking by company age

So the first major question we need to ask is: what is the impact of not tracking coverage?

Here is a quick summary of the impact:

• 68% of people who do not monitor code coverage feel they should have more time to maintain code.
• 59% of those who do monitor code coverage feel they should have more time to maintain code.
• People who monitor code coverage perceive their code quality to be 7,2% better

Conclusion: although subtle, there are benefits to tracking code coverage.
People who track code coverage feel that the quality of their code is higher, and don’t feel they need to spend more time reviewing code. This the likely reason why more mature companies, which tend to be larger and older, implement code coverage systematically.

Code coverage thresholds?

Code coverage at any given time is really just a number. It’s the evolution of that number that matters. Code coverage that goes down is an indication that new code is not being tested therefore the confidence level is going down. If it’s going up, normally it’s a good sign.

Minimum code coverage threshold to accept a Pull Request? Most companies don’t

Only 34% of companies require a minimum code coverage threshold to accept a pull request.

What is surprising is the difference between those 34% and 65%.

Here is the real difference between those two groups:

• Companies that require a minimum code coverage threshold perceive their code quality to be 6,4% better.
• Companies that don’t enforce code coverage thresholds say their biggest problem is technical debt.
• On the other hand, quality comes at a cost: companies that enforce code coverage thresholds say their biggest problem is speed of delivery.

Another important question is: what is the ideal threshold?

Code coverage popularity

The vast majority of companies that have a code coverage target have set it at 80%. This means the companies that do track code coverage actually set a very high bar with each pull request.

Conclusion: having code coverage thresholds increased perceived code quality and decreased time spent on tackling technical debt. Companies that required this threshold set the bar, on average, at 80%.

Code coverage workflow rules?

Our final consideration is about code coverage workflow rules.
Do companies require code code coverage to increase whenever new code is introduced? Or do they enforce other rules, for example requiring code coverage not to go down?

First question is: how many companies have code coverage workflow rules?

Less than 20% of companies have rules for code coverage in their workflows.

What are the rules that companies define?

We’ve asked about four major workflow:

• Coverage must be same or increase
• Coverage must increase
• Coverage must not fall
• Coverage must pass threshold

Here is the popularity of these rules:

Final thoughts

Code coverage is only as good as the quality of the tests that drive it. Setting a 100% target may tempt developers to game the system by writing unit tests that just increase coverage, at the cost of actually testing the application meaningfully. This is why the majority of the respondents reported a threshold of 80% approx, the most popular workflow rules being “coverage must not fall” or “coverage must pass threshold”.

What are your thoughts on these code coverage data points? Let me know!

Thanks for reading! If you enjoyed it, hit that heart button below or share it on the left. Would mean a lot to me and it helps other people see the facts.

You can also automate all of these points on Codacy today for free:

Click to start tracking code coverage