The True Meaning of Line Coverage

One can discuss for hours and days how much test coverage one really needs. Should it be 60%, or closer to 80%? Or should it always be 100%? But maybe it’s better to investigate for a moment what the term “line coverage” really means, and what it tells us about the system’s source code and unit tests.

Line coverage is usually measured as follows: all automated unit tests are run against the source code, and every line of source code that is executed at least once during the test run is counted as “covered”. This number is than divided by the total number of executable lines of source code, resulting in a percentage that is hopefully closer to 100% than 0%. For branch coverage, the same exercise is done, but counting branches instead of lines of code.

There are several problems with this way of measuring test coverage. For example, not all tools verify that the unit tests conclude with an assertion or an expectation for an exception to be thrown. Except for some special cases, without an assertion, nothing is really tested, and the lines and branches executed during the unit test shouldn’t be counted as covered.

But even if there’s an assertion present, the assertion doesn’t always verify something that has something to do with the source code that’s being tested. Maybe the assertion just verifies that after the source code has been run, 2 + 2 still is 4. Therefore, the presence of an assertion alone isn’t a sufficient condition to conclude that the lines covered by a unit test really have been tested. The only thing we know about these lines and branches is that they didn’t stop the unit tests from finishing successfully. And that’s something completely different than being properly tested or correct.

Just to make it clear: if a system has a test coverage of e.g. 70%, that really means that

• 30% of the source code isn’t executed at all by the unit tests,
• 70% of the source code isn’t stopping the unit tests from running successfully,
• we do not really know whether the unit tests do anything meaningful or not.

Of course, if the developers have cared to write unit tests that cover 70% of the source code, we know from experience there’s a good chance that the unit tests are doing something meaningful. Most probably, the source code is in a much better shape than in projects without such a level of test coverage. A quick inspection of a small number of unit tests will often be enough to confirm that, or reveal the contrary.

However, if you want harder evidence that the source code is really being tested by the unit tests, you should use other techniques and measures than unit tests and their test coverage alone. Mutation testing is one such technique, and it’s a very powerful one too. Of course, it’s not entirely fool-proof either, because in the end, everything depends on having good developers in your project team. But writing meaningless unit tests just to kill a few more mutants is far too much work for lazy developers, and often too difficult too. It’s one of the best guarantees I know to ensure that the unit tests are of good quality.