Broken promise of 100% code coverage
100% code coverage… It’s a dream of a developer, isn’t it? Your code is fully tested, it is correct and there is nothing to worry about. Your code can be deployed now and you don’t feel anxious about it. Is it true?
Code coverage is frequently used to measure quality of unit-testing. It is often the metric for that. This fact is not surprising at all: the value is simple and easy to understand, it is easy to calculate and easy to quantify.
The fact that it is easy to measure code coverage doesn’t make it a good metric though. You can get into trouble even if your code coverage is 100%.
Let’s take look at an example.
We have a very simple method…
…and a test for it:
This unit test produces the perfect 100% test coverage for the elementAtIndex: function.
Does it prove that the function works correctly? The answer is, obviously, no. What happens when we exceed the array boundaries?
Why did that happen? When you try to focus on the code coverage metric you write code that looks at the implementation of the tested function/method. But the implementation is not proven to be correct yet. That is the reason why we want to test it.
Even with that simple function code coverage failed as a good metric to measure the quality of unit-tests.
What to do instead? Don’t look at the actual implementation of the method, look at the contract instead. Look precisely at the outputs of the function/method for any specific inputs. Look at the side-effects that this function does or uses. Take into account the possible edge cases that might exist. List this information and make tests according to that.
Let’s get back to the elementAtIndex: method.
There are 2 types of possible input:
- Valid input, value exists. We need to check the validity of value.
- Index we send in is out of bounds.
There is also 1 side-effect that is used: the _data array.
That test suite is written using our expectation of the function behaviour and it is nothing to do with the code itself. It shows that our code was not correct, and we have to fix it.
That was a very simple example. Using it I want to illustrate a very important point:
100% code coverage doesn’t mean that your code is 100% correct.
When you write unit-tests for your code, don’t focus on achieving perfect code-coverage. Focus on testing 100% of the contract of the function.
See also:
