Approval Testing

Have you ever written a test assertion like assertEquals("", functionCall())? Where functionCall() is returning String and you’re not sure exactly what that string should be, but you’ll know it’s right when you see it? When you run the test the first time, of course, it fails because functionCall() returns a string that isn’t empty. (You might have several tries, until the return value looks correct.) Then you paste this value instead of the empty string in the assertEquals(). Now the test should pass. Result! That’s what I’d call approval testing.

The crucial step here is when you decide the output is correct, and use it as the expected value. You ‘approve’ a result — it’s good enough to keep. I expect you’ve done this kind of thing without really thinking about it. Perhaps you call it by a different name: It’s also called snapshot testing or golden master testing. In my experience, if you have a testing framework specifically designed to support it, then a lot of things fall into place and testing this way gets easier.

With a classic unit testing framework like JUnit, it can be a bit painful to update those expected strings when they change. You end up pasting stuff around in the source code. With an approval testing tool the approved string gets stored in a file instead. That immediately opens up new possibilities. You can use a proper diff tool to go through changes and merge them one by one. You can get syntax highlighting for JSON strings and suchlike. You can search and replace updates across several tests in different classes.

So, what are good situations for approval testing?

Code without unit tests that you need to change

If the code is in production, then anything it does is, by default, considered correct and can be approved. The hard part about creating tests turns into a problem of finding seams and carving out pieces of logic that return something interesting you can approve.

REST APIs and functions that return JSON or XML

If the result is a longer string then storing it outside the source code is a big win. JSON and XML can both be formatted with consistent white space so they are easy to compare against an expected value. If there are values in the JSON or XML that vary a lot — dates and times, for example — you might need to check them separately before replacing them with a fixed string and approving the remainder.

Business logic which builds a complex return object

Start by writing a Printer class that can take your complex return object and format it as a string. Think of a Receipt or a Prescription or anOrder. Any of those could be represented well as a human-readable multi-line string. Your Printer can choose to only print a summary — traverse the object graph to pull out relevant details. Your tests will then exercise various business rules, and use the Printer to create a string for approval. If you have a non-coding product owner or business analyst, they could even read the test results and verify they are correct.

If you already have tests that make assertions about strings that are longer than one line, then I recommend finding out more about approval testing and to start using a tool that supports it.