Usability testing: Math behind magic number 5
Most of us in the design community are pretty much familiar with the concept of usability testing. One of the methods involves conducting user interview sessions with a targeted group of users in order to test the prototype of a product. Naturally, a lot of company’s resources is spent in the process which is directly proportional to the number of users tested with.
In this article I’m gonna address the question of “How many users are enough for user testing?”.
What is magic number 5?
Through his article, “Why You Only Need to Test with 5 Users”, Jakob Nielsen very boldly suggests that using no more than 5 users for a user testing session can give us the best results. And using any more will only result in a waste of resources.
The basic idea is, 5 users can uncover most of a product’s usability problems.
Math behind the magic
Jakob Nielsen & Tom Landauer wrote a research paper together which gave us the following relationship between the number of users and the proportion of all problems found;
Where p is the probability of detecting a given problem. The value of p is taken to be 31%, averaged across a large number of tests conducted (data available on the research paper).
n is the number of users & N is the number of total problems in a test session.
By this relation we find that 5 users can uncover approximately 80% of the usability issues.
But what about the remaining 20% ???
Well, increasing the number of users will clearly make us reach closer to that 100. But note that the graph is asymptotic, it will never really reach there. Plus, the decrease in slope suggests that eventually, no matter now many users you add, there will be very insignificant growth in the percentage of usability problems to be found.
Lost you there? Let me explain.
Testing with 1 user will uncover approximately 33% of the total problems.
Testing with 2 users will uncover more than 50% of the total problems.
.
.
Testing with 5 users will uncover more than 85% of the total problems.
Testing with 6 users will uncover about 87% of the total problems.
The increase in percentage is less for higher number of users.
At this point there’s a high chance of getting repeated insights. Given that our users are humans, we can’t expect most of them to uncover that remaining 10–15 percent of the total problems. We might not find that Nth user ever, and the amount of resources spent will linearly increase giving us no new information.
Should 5 be the absolute number?
NO! The point of this article was not to restrict your mind into thinking this approach is the best. I wanted to explain the math behind this concept in order to show that you don’t need 50 users to test a product.
You can definitely go further than 5 but you must stop once you see most of your insights repeat!
Don’t have a linear approach thinking more users will give more insights. Because number of users and number of problems are not directly proportional, which we can see from the graph itself.
Is this equation helpful in the real world?
Hell yes! Here at OYO, I had to conduct multiple user interviews to test out the design 2.0 version of the consumer app. And I myself observed the insights getting repeated a lot after the 5th or 6th user. Most of my tests involved 7–8 users at max but the insights I had received were tremendously useful.
This equation helps us understand when to stop worrying about the number of users and start thinking about other factors that can help us get more insights.
Other factors to help make best of the user testing
- Focusing more on the test design and complexity of the tasks given to a user.
- Targeting the persona very thoughtfully.
- Analysing the results after each user testing to look for repeating insights or patterns that may or may not seem familiar.
- Using better tools for conducting sessions and analysing results.
- Looking out for faulty data and biased feedbacks. In case you feel that a certain user was being biased with the feedbacks or the test environment was not very ideal, learn to discard that data and replace with a new user with same persona.
Thanks for coming this far! I hope this article proves useful.
