Test Coverage vs Requirements Coverage vs Code Coverage

Welcome! Next 10 minutes we’ll see the approaches to measure various kind s of coverages and what’s the differences between them. For sure we will do it by example as I usualy do to make the problem more obvious and focus on practicing. Have fun while reading it.

Test Coverage

Definition. Test Coverage is one of the quality metrics that represents the requirements part covered by tests (or test cases). In case when the only requirement source is application code we can say that Test Coverage represents the lines of code covered with tests (or test cases).

How it can be used. To identify which parts of application need more attention, more coverage. For example, the critical path for clients should be covered as much as possible and non-critical creating test cases for non-critical paths can be postponed. We can also estimate the percentage of successed tests in the following release and decide if we’re ready to deploy.

The disadvantages of this coverage:

• We still cannot identify if every requirement of application is implemented. Such analysis should be done manually and non-existed test cases should be added to the test set
• We don’t know if we’re covering all the code lines in application. As we’re focusing on functional requirements and we do not know the details of implementations we cannot garantee that all executed test cases covers all lines of code

The example. Let’s say we have bank terminal and user can request money from it. The algorithm is the following:

• If user requested < $100 then commission $5 should be applied and total amount should be increased by $5
• If user has insufficient balance then transaction should be declined and no money given
• After money is given the balance decreased by Total amount

To cover all of these requirements we can write the following test cases:

• User have balance $500 balance. Try to take $100 => PASSED ($400 remain)
• User have balance $500 balance. Try to take $99 => PASSED ($396 remain)
• User have balance $500 balance. Try to take $501 => FAILED
• User have balance $99 balance. Try to take $95 => FAILED (not enough money to cover the commission)

This test cases cover all the requirements we have. Is it enough to say we have 100% Test Coverage? Let’s see it in the further sections.

Requirements Coverage

Definition. Requirements Coverage can be measured by the following formula:

Tcov = (Lcov / Ltotal) * 100%

where Tcov is a test coverage, Lcov — number of requirements tested by test cases, Ltotal — number of all requirements in the application.

How it can be used. If we’re using TMS (test managements systems) we can look on the requirements coverage. The methodology is pretty simple: we have to track every requirement and every item should be tied with a test case or a set of test cases in TMS. Now we can measure the Requirements Coverage.

The disadvantages of requirements coverage:

• If we do not use the relations between requirements and test cases we cannot measure the requirements coverage. So we need to be careful and reserve the time for such activities
• There can be “white spots” — the requirements that are not covered with tests and thus we cannot say if this functionality is implemented
• Need to review test cases by daily basis. When requirement is changed test cases for this requirements becomes useless, they verify nothing anymore. Such test cases should be removed from the test set and for sure we need time to identify such cases.

The example. We got the Requirements Coverage in previous section when test cases for functionality were designed.

Code Coverage

Definition. The code coverage can be measured by the following:

Tcov = (Ltc / Lcode) * 100%

where Tcov — test coverage, Ltc — number of code lines covered by tests, Lcode — number of code lines in total.

There’s a bunch of tools in our days that can automate this work for you and to measure the code coverage by automted tests. We’re not focusing on them so it might be good topic to discuss in the future.

How it can be used. We can see in real time what lines of code were covered by automated tests, which ones should be covered in future and which should be removed as duplicated ones. The main reason to measure Code Coverage is to improve your development velocity and stability. There’re a lot of articles dedicated to “unit test effect” so you can find them and verify. This can be done by the obligatory rule not to merge code into main branch if the coverage is less than 80% for example, in other words to create a quality gate.

The disadvantages of code coverage:

• You’ll never know if all the requirements are covered using only this metric
• Using code coverage analysis with black-box testing requires a lot of effort for developers and QA guys. However, this method works well with white-box testing techniques like unit ot integration tests.

The example. Let’s say I’m the developer and I have written a class responsible for giving money to user and unit tests as well:

As you can see here I’ve made a mistake and forgot to check the current user balance so I can easily get the negative balance after money was given. But my code coverage is 100% and I can proove it:

Code Coverage report from Intellij IDEA tool

Consclusions and next steps

Today we’ve learned about various coverage options. Test coverage is a general thing that can be measured by 2 way:

• Requirements Coverage if you have requirements as a source
• Code Coverage if your source is code only

By measuring all the coverages we can see the full picture of our developments processes and identify the weaknesses to eliminate. You can also explain the differences between the coverage measurements to show your developers that “we have 100% unit tests coverage” approach is not enough to garantee all application functions work well and there’re a lot more points to collaborate.

Feel free to join me on telegram channel to get more content related to QA Automation.

Good luck and don’t stop to automate!