Many Facets of Unit Testing
Tales of Software Engineering — 3
Unit Testing is not what you think it is. Or at least, not only. Usually Software Engineers think that Unit Testing has the purpose of cursorily testing their freshly written code. It is not rare to hear that Unit Tests should be very minimal and just make sure that the function or module at hand will not crash when called with one example of input. In some Software projects, there is even the belief that Unit Tests are actually a burden. The argument goes like this: the Software will need to be refactored anyway, and the Unit Tests will be thrown away as a result, so why bother? In this post, we will look at the many facets of Unit Testing, which in fact go well beyond just testing a function or component.
Engaging with the problem
Unit Testing is first about engaging deeply with the problem at hand, and the code you are going to write. Yes, that is right, the code you are going to write, future tense. That is because when presented with functionality to deliver, provided it is not trivial (in which case you would re-use, not write new code, right?), it is very useful to first explore and play with the problem. That can be done very efficiently by writing down a few test cases, and playing with them in an interpreted environment like ipython. In that stage, it is useful to start systematically collecting test cases and even to encode them in a burgeoning test harness. These test cases become a form of specification, and of course, validating them with your clients (which could be just you if you are writing e.g. a function deep inside a library) is a must.
Note that Unit Tests presuppose that the whole system under consideration can be decomposed into clearly identifiable parts. These parts can be functions, classes or whole modules, but for the purposes of Unit Testing, the smaller, the better. If those parts are hard to separate, then it can be an indicator that the design is wrong.
Armed with a simple Unit Test harness, writing code becomes more focused and efficient: the code will converge faster to producing the needed results. There is a clear goal, and the verification of that goal has been mechanized. Either the Unit Tests pass or they don’t, and you can run these Unit Tests to verify your code a hundred times a day if you need to. Some tests will prod you to understand your code more deeply, why it doesn’t work, and how to make it work. It is a great way to challenge assumptions that might be buried in the code more or less implicitly. Is is also a very interactive process, an interplay between the tests and the code, where the tests make you think more about the code you write, in a guided way, and where the correct design and code emerge in response to specific requirements. In that phase, it is very common to modify APIs to make them more comfortable for the end-user — which could very well be yourself.
At the end of this stage, when your code provides the expected functionality on average and has comfortable and efficient interfaces, it is a good point to ask yourself if you can do better in terms of simplicity. Can you now pass all these Unit Tests with even less code? The Unit Tests are crucial to make this first refactoring even possible: they will make sure the new, improved code delivers the expected results. Of course, you can iterate a few times already, replace those maps by smaller sorted arrays, or parameterize classes that you had initially hard-coded to handle only integers.
A certificate of correctness (and performance)
As you write more code, and it passes more and more of the Unit Tests you have set up upfront, the game is now to beef up the Unit Tests by including all sorts of corner cases. The Unit Tests become a kind of certificate of correctness for the code. Asking yourself what those corner cases even are, and how to handle them, should lead you to review and revise your code, again engaging deeply with everything that happens in your function or class. It is best here to adopt an adversarial mindset, and to try and break your code … till it doesn’t break anymore. Now is a good time to write a test for every weakness in your code (and you know those, because you wrote the code!), and of course, improve the code till these new Unit Tests also pass. This is a again a very interactive process, a continuation of the initial process we outlined above, this time with a focus on correctness more than functionality.
An important aspect of this certificate of correctness is that bugs are found earlier in the development process, by the developers, who are the most likely to be able to understand the issues and fix them quickly. Bugs that sneak through Unit Test “walls” will be costlier to track down and fix. Unit Tests thus reduce overall development costs (and improve development velocity).
In some limited situations, Unit Tests can be as powerful as a formal proof of correctness. But this can unfortunately be true only when testing scopes that are quite limited. In those cases, it might be possible (not always) to exhaustively exercise ALL the code paths and therefore provide a “complete” certificate of correctness. This however breaks down if the scope tested is too big, or if there are too many edge cases to consider. Conversely though, if the certificate provided by the Unit Tests is too far from being “complete”, it could be a sign that the scope considered is too big.
As a side note, conscientious Software Engineers will routinely spend 3 to 5 units of time testing for each unit of time they spending writing code that goes to production. There is little downside to spending more time on Unit Testing, usually. Even if the Unit Tests have to be maintained when the code is refactored, in the process of maintaining them they will probably ask pertinent questions about the new code. Of course, some Unit Tests will be thrown away. From which it does not logically follow that it is not worth writing Unit Tests, as you often hear.
Enabling maintenance and evolution
Unit tests are a requirement to be able to maintain and evolve code. Here is a scenario that is not so rare, unfortunately: some code base was written by a team and has gotten pretty complex as it was put in production and had to handle more and more corner cases. It was not tested appropriately (besides being “tested in prod”) by the “previous team”. Then comes a new requirement, or a migration. But the current team is paralyzed and unable to evolve that code base, because they cannot tell if they are going to break something or not. There is no safety net of Unit Tests to guide the changes they want to make. If they do attempt to evolve the code, it will be excruciatingly painful. The cost is of course time to release, but there is also a human cost. Developers will not enjoy the exercise (except for a few who like to fix things as a personal trait), so they might start thinking about leaving the team. Or even the company. Code that does not have a good hedge of Unit Tests is not maintainable and not refactorable, or at a cost that destroys the project’s velocity.
Documentation
The Unit Tests serve the role of specification and then of certificate of correctness. But they are also an important documentation artifact for the code. Someone — including yourself in the future — can now look at the tests, and understand what is going on in the code, on average and for the corner cases too. Unit Tests should be thoroughly documented, to further help the reader understand the tests faster. Writing the code is not complete till comprehensive Unit Tests are written and documented. Unfortunately, this a lofty goal that few organizations will ever achieve, partly because of short-term economic pressures on the Software development process.
Conclusion
There are situations where spending a lot of time on Unit Tests might not make sense. For example, the code could be prototype code that will be thrown away soon. Or writing comprehensive Unit Tests might be hard or not worth it from an economic standpoint, if it is impossible to find small enough units to test, or if there are too many corner cases. Unit Tests are not the only way Software Engineers can improve the quality of their systems either. There is also Integration Testing, which focuses on the interactions between modules of Software. But when done properly, Unit Testing is a powerful and in fact unavoidable tool to ensure not only correctness, but also velocity, both when writing the initial code, and in the face of constant refactoring.
