Test Desiderata

Kent Beck
Kent Beck
Oct 18 · 2 min read
System under test

Go placidly amid the noise and haste,
and remember what peace there may be in silence

Recently, I published this tweet:

Tests should be coupled to the behavior of code and decoupled from the structure of code. Seeing tests that fail on both counts.

I thought this property of tests was obvious and widely understood, but the tweet blew up. I have seen tests which are just a horrible syntax reiterating exactly what is already said in the source code under test. I went and re-read TDD: By Example and these properties are nowhere to be found, so I guess I shouldn’t have been surprised.

The experience got me thinking about the other properties of tests that I take for granted. Here, then, is a list of properties of tests. Not all tests need to exhibit all properties. However, no property should be given up without receiving a property of greater value in return.

Properties

  • Isolated — tests should return the same results regardless of the order in which they are run.
  • Composable — if tests are isolated, then I can run 1 or 10 or 100 or 1,000,000 and get the same results.
  • Fast — tests should run quickly.
  • Inspiring — passing the tests should inspire confidence
  • Writable — tests should be cheap to write relative to the cost of the code being tested.
  • Readable — tests should be comprehensible for reader, invoking the motivation for writing this particular test.
  • Behavioral — tests should be sensitive to changes in the behavior of the code under test. If the behavior changes, the test result should change.
  • Structure-insensitive — tests should not change their result if the structure of the code changes.
  • Automated — tests should run without human intervention.
  • Specific — if a test fails, the cause of the failure should be obvious.
  • Deterministic — if nothing changes, the test result shouldn’t change.
  • Predictive — if the tests all pass, then the code under test should be suitable for production.

Combinations

Here are some attractors in the space of all possible tests:

  • Programmer (aka “unit” tests). Give up tests being predictive and inspiring for being writable, fast, and specific.
  • Acceptance. Make tests readable for non-programmers even if you give up some speed and specificity.
  • Monitoring. Yes, monitoring is a test too, or rather tests and monitoring are both forms of feedback. Monitoring abandons being predictive and to some degree being automated (modulo alerting).

So What?

Look at the last test you wrote. Which properties does it have? Which does it lack? Is that the tradeoff you want to make?

Kent Beck

Written by

Kent Beck

Kent is a long-time programmer who also sings, plays guitar, plays poker, and makes cheese. He works at Gusto, the small business people platform.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade