Test Coverage vs Code Coverage
Please do watch video (https://www.youtube.com/shorts/G0Q5-qvCfhQ) to start our journey to explore more on Test Coverage and Code Coverage …
Is 100% code coverage sufficient for deploying our changes/features to production faster, whereas having test coverage will slow down the progress?
“Code Coverage and Test Coverage ‘’, the most common and important terms used while delivering/working towards software development cycle. I was first exposed to these terms, during one of my software engineering lectures and I admit that I considered these as the same terms by a basic concept or misconception i.e. how can our tests that cover the entire code base miss any functional part in the application?
Now after spending significant time in the software industry (working as a tester to a SDET, to an Architect, to a Program Manager and Capability/Chapter lead), I often observe the same level of confusion still persists among most of the engineers (including devs and testers). I have been part of multiple discussions around similar lines e.g. with XX% code coverage we do not need to maintain and track tests coverage from manual tests as it may slow down the deployment speed. I am writing this post to share my personal experience and learning, which really boosted my and my team’s confidence to deliver multiple enterprise products/solutions with optimum quality frequently to production.
To embark on the journey, let’s start by answering the following questions:
- Are Test and Code coverage same?
- If not the same, which one is better and why?
Are Test and Code Coverage the Same?
Though these two coverage i.e. Test and Code sound similar but their root purpose is quite different, one is more focused on “quantity’’ whereas the other is oriented towards the “quality” aspect.
Code Coverage:
This is the measurement of the code you have covered through your tests. If we have our tests which hit our 50 lines of code out of say total 100 lines of code, then we have a 50% code coverage. Whereas, if we are covering 100% of the code, we can say that we have executed each line of the code and it has not give any errors.
So the question that arises here is, with 100% of the code coverage, are we confident enough to provide test sign off to confirm that code base adhere to optimum quality? Let’s hold our horses here… and discuss the other companion i.e. test coverage.
Test Coverage:
The code coverage part helps you analyze if each code in the application is being executed by our tests or not. Test coverage is a bit different. The test coverage defines whether our tests are covering the functionality of the application or not.
So, if we have 100 test cases and 80 of them got executed while testing the application functionality, it means we get 80% test coverage. If you are looking to increase the number, you just have to look at the functionality that you apparently missed and write test cases for it.
Observed the difference between these two?
We see from above that 80% code coverage and 80% test coverage can mean totally different things in software testing. Hence, using them interchangeably is a bad practice that can lead to misunderstanding regarding the quality of the software.
Test Or Code Coverage — which one is better and why?
Let’s quickly analyze some characteristics with definitions of these two terms:
Finally let’s take a look at which one (Test or Code Coverage) is better and our previous question i.e.
With 100% of the code coverage, are we confident enough to provide test sign off to confirm that codebase adhere to optimum quality/expectations?
As per my personal experience in delivering quality enterprise products to our esteemed customers across various industries and domains, considering only 100% code coverage will be an outlier and can lead to undiscovered defects.
To demonstrate this, let’s take a basic example, where the following function that we have implemented in first iteration:
public bool calculate(int firstNum, int secondNum)
{
return firstNum * secondNum
}
To test this we have written following code:
public void testCalculation():
{
assert.IsTrue(calculate(1,1) == 1);
}
The test achieves 100% coverage above. Now let’s say the function’s operation is changed to division instead of multiplication inadvertently. This test would still pass in that case. Hence, 100% code coverage has no relation to the code quality or software quality. Solely relying on this could mean passing on bugs to production that are fatal for the software.
With that being said, code coverage does help in determining if each line of code executes, and if there are any bugs in the branches or syntax etc., they can be easily pointed out.
Therefore, it is alright to keep the number higher (80% maybe), but writing test cases just for achieving 100% code coverage should be avoided. If you get 100% code coverage anyway, it is well and good for all.
Final Takeaway
With the details above, we can conclude that, except from having similar names and core requirement for having tests for ‘Test coverage’ and ‘code coverage’, there is nothing common in these two terms. Code coverage just focuses on how many lines we have covered by our unit tests, whereas Test coverage is a bit different and focuses purely on the application’s behavior and finding defects.
Both contribute in their own way to the software quality. These should be used in a way that compliment each other and we should work collaboratively on how much time we should give to each of them. If we just perform test coverage on a branch and not the code coverage for instance, you might miss the bug in another part that could affect the functionality.
With that being said, in my personal opinion and experience, instead of targeting 100% code coverage as primary measurement to deploy we should rather work on making a well collaborated decision to combine and mix both test and code coverage to get optimum confidence with quality to deliver releases faster.
Special thanks to Garima Uppadhyay Raj Puttakota for taking sometime out to review the content and providing valuable feedback.