Is 100% tests coverage a holy grail of software development?
What counts for tests coverage?
There are many things that count in tests coverage report, statements, possible branches, functions and so on. But it’s still missing one of the most important things — the quality of the tests. There are no “tests for test” and moreover report generating tools are unable to measure tests accuracy and quality.
In fact you can achieve full tests coverage in the reports with useless tests. But as a stakeholder, when you hear that software is 100% covered by the tests, probably you will start assuming that it’s working as expected in every possible scenario. I’m afraid that you might be disappointed.
When you decide to push your application live, you have to take into account that you are introducing multiple ways for your software to fail. Fails that you couldn’t predict. Specialists involved in software development process such as engineers, product owners and managers, know how application should work and how they want it to be used. However, it’s safe to say that none of them can predict every use case. The number of possible users is almost unlimited and each of them can use your software in a different manner. It can be something irrelevant or something incredibly clever. Users can guess out processes under the hood of your application and hack it in a way that no one have ever thought. Also it’s worth to keep in mind that software itself is nothing without hardware… which can fail as well. It’s impossible to cover all of possible use cases nor with Unit, Integration, Interface, Manual and any other kind of tests.
The only way we can play with it, is to figure out as many probabilities of usage as we can.
As it was mentioned before, when you develop piece of software, you know expectation of customers and business. Those are requirements. Each business requirement should be covered by tests with positive and negative scenario, based on test data and actions. When you are done with this, it’s time to put on user’s shoes and try to figure out as many ways of misusing application as possible and write test cases for it. I’m pretty sure there is always another test that could be added. And as you can see now, 100% tests coverage can’t give certainty for software. It can build confidence and reduce number of issues, but never gives certainty.
You can achieve code coverage up to 100%, but it will only guarantee that your software is working as expected in scenarios covered by your tests. Not in every scenario.
Should we always aim for 100% tests coverage? In my opinion couple of those percentage points should be discussed. I don’t think you should always keep your eyes on tests run output showing perfect 100%. There are many cases where it’s unachievable or not reasonable.
Not every single software product is covered by tests. And you can join the team which is working on such for a long time. Then aiming for 100% tests coverage sounds like a Mount Everest of software engineering. In such situation, one of possible approaches is to set smaller checkpoints on the way up to the hill by measuring tests coverage, and putting up minimum required coverage level step by step. This can give a good feeling of achievement and will make you closer to the pike.
If you are using frameworks or libraries, it’s highly possible that those were tested by their authors and you don’t have to test them by yourself. Even that it’s still a part of your codebase and have real impact on your application.
When you already have very high tests coverage, around 85%, you shouldn’t think about putting up tests coverage results. This way, you are priming your mind to think out a ways of hacking metrics generator to show better results. Instead, you should think about every possible test case that can bring value and quality to your application. Even without improving statistics.
We have to remember about code which is difficult to cover in unit tests as file system manipulation or web services calls. Those are usually mocked in unit tests and assumes that everything there “in the other side” is working as we would like it to work. What is not always true.
Wrapping everything up, it’s easy to say that 100% tests coverage is not a holy grail in software development. It shouldn’t be our goal. It is proved that it’s easy to hack metrics generators. Coverage reports are not showing real quality and value of our tests. We should aim for covering every use case of our applications which we can figure. Even without improving statistics.
