Understanding the enigmas of usability testing

How probability demystifies sample size, personas, and communication

I vividly remember interviewing a participant for the first time. Riddled with leading questions, it was a subpar experience for all parties involved. By the fourth participant, something hard to explain happened, I could predict what the next person would do.

‘This is a common phenomena’ said my manager, who confidently booked 5 participants for the test. Incredible, how on earth did they know this was all the people we would need to get results?

When asked about the origin of these oracle-like abilities, they pointed me to the much cited Nielsen Norman Group (NNG) article ‘Why you only need to test with 5 users’ which is based off the 1993 published paper ‘A Mathematical Model of the Finding of Usability Problems’.

The now seminal paper, aggregates 11 usability tests and heuristic evaluation studies to assess the chances of finding all usability errors. They focused on budgets, time and cost to find the best way to run these tests. Nielsen et al found that 4 to 6 people is the best cost against outcome with diminishing returns at anything above 16 people.

This felt like a parent replying to an inane question with ‘it works just because’. The study…