“The Biscuit Factory Model” for test sampling

How much do you trust your automation? It’s actually a very uncomfortable question to ask!

As testers we rely on automation to make our lives easier, especially in agile teams. We just don’t have time to test everything every sprint.

Likewise we all have a tale about an automated test suite which on examination “wasn’t doing what it was supposed to”. We’ve all been blindsided — for myself there was a test which we’d had for years would renew an application, but was actually missing the final “submit” and associated check. Sadly of course, we only realised this was a problem once an issue occurred in production.

But you have to trust your automation to some point. If you are repeating everything you’re doing in automation, you’re not getting the value. What do you do?

I use a model for this that I call “Biscuit Factory Testing” — which I use both for automation and testing done by third parties.

I used to work in a biscuit factory (you Americans might call them “cookies”) during the Summer. How do you test the quality of biscuits? It’s not like you can take a nibble out of every biscuit on the production line.

In the food industry, they’ll typically take a batch, and within it they’ll take some random samples, and check them in detail. This random sample will represent the much bigger group. If there’s any issues, they’ll escalate, and investigate the whole batch — sometimes binning the whole lot, and investigating what’s wrong. It throws a warning light.

Likewise in testing, I like to choose a few random tests, and repeat them myself, also going into detail of what’s been logged. It needs to be random — as in time you will be able to view the whole suite and hopefully unearth any issue.

I mentioned this model during Q&A as part of today’s Automation Guild! Hopefully you signed up, it was a lot of fun.