Uncertainty-Driven Testing

Understanding what TDD actually is about…

I introduce here Uncertainty-Driven Testing as a method for clarifying what we test and why. Good developers identify what they don’t know, and use tests to reduce uncertainty. This is only my opinion and experience, please agree or disagree but make sense of it.

What are tests?

Traditionally, tests aim at checking, through concrete execution examples, whether an implementation meets its specification:

1. The test subject is a callable P (a function, method, program, web service, whatever) with its specification S. The latter is a description of P's INPUT and OUTPUT, together with PRE and POST conditions on them. Note that P's internal state can actually be modelled through INPUT and OUTPUT and can be ignored without loss of generality.
2. A test case invokes P with some input, and checks whether its results meet expectations on OUTPUT and POST, through so-called assertions. If P is robust then a robustness test case invokes P with input violating the PRE on purpose and checks whether the result is as expected (typically an exception being raised).
3. A test suite is a collection of test cases that together provide enough confidence that P's implementation is correct with respect to S. The test suite is not a correctness proof, though.

This is how tests are traditionally explained in software engineering. I’ve observed that Test Driven Development (TDD) and Behaviour Driven Development (BDD) have brought some confusion, because they don’t exactly fit that traditional model. I propose here Uncertainty-Driven Testing as a broader framework to reconcile traditional and agile visions of testing.

Why do we test?

I mentioned that a test suite provides confidence about the implementation correctness. To cover more testing use cases, though, we can abstract a little bit over that motivation, saying:

Tests aim at reducing uncertainty

With such a broader definition, we can now look at specific testing use-cases, and see where uncertainty hides:

• Traditional testing (explained above) aims at reducing uncertainty about the correctness of P's implementation against S, under the assumption that S is known. In such a scheme, tests are often written after the implementation.
• In Test (and Behaviour) Driven Development, tests are written gradually and drive the implementation. Those techniques act that S it often not well known or is incomplete. Hence uncertainty lies in the specification itself and the techniques help building both S and P inductively through examples and (possibly) counterexamples.
• In Test (and Behaviour) Driven Design, the specification may be known and complete, but the developer faces uncertainty about the solution design. An initial implementation is challenged one (counter)example at a time, making the design emerge as complexity increases with new test cases.

Other testing use cases exist:

• When using a third party reusable library or method for the first time, I frequently write automated tests to check my understanding of its interface and behaviour. This is indeed where uncertainty is.
• Some tests aim at providing rail guards against future changes that might introduce bugs (regression testing is somewhat of that kind too). Uncertainty here is about the team ability to make changes that are fully backward compatible with S.
• Pending tests (tests whose failure is expected but do not prevent the test suite from succeeding) can be used to track uncertainty about the specification in very specific corner cases, uncertainty about a specific bug fix.
• When I coach a junior developer through pair programming, I sometimes use ping-pong programming where I write a first failing test, then the developer writes the implementation and the next failing test, and so on. My aim is to check whether we will make a good pair, and whether the junior knows that both specification and tests help. Simply because I don’t know him/her very well yet.

How?

If you agree with my arguments, then the way you write tests must be driven by uncertainty (Uncertainty Driven Testing, or UDT). The tests you need are those that make you more confident about something. Some interesting consequences:

• Required tests depend on the developer: seniority and experience facing a given task, programming style, language, framework, library, etc.
• Required tests depend on the implementation quality: when P's implementation is ugly, you'd better add more tests. Your reasoning abilities about correctness are limited, hence uncertainty is flying high, especially if the code frequently changes.
• Required tests depend on the type system: that Haskell developers need less tests than Ruby developers seems quite obvious to me (provided they are senior, of course, see previous bullet), at least if we are specifically talking about correctness uncertainty. A great type system provides you with a lot of certainty already, because it removes many invalid execution paths.
• Required tests depend on the programming style: declarative programming requires more upfront thinking, hence less tests. That’s because the program P and specification S tend to become one, naturally reducing the possibility for them to diverge.
• Required tests depend on the specification quality and completeness: TDD and BDD provide the ultimate example. When no specification exists, writing examples of what you want and don’t want is certainly a good way to start. That’s what inductive learning is about, and you’d better not underestimate how much our brains love it.

With experience and seniority (that depend on your comfort zone), you will discover that low-level tests (e.g. unit tests against private interfaces) are not that great, and sometimes a pure waste of time. You should instead write tests against public and stable software boundaries exposing non trivial specifications. Regarding both correctness testing and specification completeness, that’s where your testing effort pays off.

Do you enjoy this writing? Please follow me on Twitter, hire Enspirit for your software project, or use Klaro for tracking your processes.