On Building Software

Most Bugs Live in the Permutations

Unit tests test for code at the very foundation. Each little possible isolated bit of functionality. Test for all these functions in isolation and you may feel confident in your code.

Your full count of tests for a particular area of code is the summation of these tests.

3 + 6 + 3 + 4 + 2 + 3 + 2 + 5 + 4 + 5 = 37

In reality, the number of potential pathways through related tests is more like a multiplier of each unit test grouping: Every possible combination of these tests grouped by the section of code they are testing.

3 x 6 x 3 x 4 x 2 x 3 x 2 x 5 x 4 x 5 = 259200

I’ve found nearly all bugs live in the combination of these unit outcomes — in the leaks through the pathways.

Unit tests are good for you if you understand their limitations. But if you listen to unit test zealots, they tend to overvalue their reach. I’ve found spending that amount of time on those sorts of tests is a penny-wise, pound-foolish thing.

Much more attention should be placed on the pathways.

Of course, you aren’t going to write “259,200” integration tests. And even if the vast majority of these combinations don’t exist as actual pathways (say only 5% of those pathways are even possible), the number is still much much higher than the number of unit tests you’ve written.

259200 x 0.05 = 12960

You still can’t practically write tests for all this. And so, confidence must be found in other ways.

The way I find confidence is simply by spending a lot of time on one system. Getting intimately familiar with it. All of its pathways. All of its rough edges.

Because familiarity with a system is the most critical component of having confidence with the system.