Are five participants really enough for Usability Studies?
Five participant Usability Studies have become really, really, common in the field of UX Research. When I first entered the UX field back in 2010 I was quickly passed the Alertbox “useit.com” article from Jakob Nielsen that draws the conclusion that by testing with five users you can find 85% of the usability issues in an interface.
The team I was on ran lots of these small studies because they were cheap, quick, and often insightful. We made a lot of impact with them and fixed many problematic designs before they got into the hands of users. Today, there are tons of variations on these studies (in-person, remote unmoderated, RITE, the list goes on…).
Where it originates
Neilsens work looked at the proportion of usability problems discovered while testing a single user across lots of studies. Theres a formula he developed and socialized in his article “N(1-(1-L)n” where N = the total number of usability problems in a design and L as the proportion of usability problems discovered while testing a single user. It’s a variant of the cumulative binomial probability, that other researchers before him studied using. Neilsen found the typical value of L is 31%, a value he arrived at after averaging this value across a large number of projects. He visualized this in a graph most UX Researchers know well…
Behold…As you move up and to the right by testing with more people, the number of usability problems found levels off. The argument goes that theres most to be gained by testing with the first few participants, while the latter two thirds of participants providing you with increasingly fewer issues.
An important point I want to make is that his studies and those that his work references don’t actually suggest that the five-user sample size is the most effective way of locating all the usability errors in an interface, and sometimes you can miss some bad really bad ones!
In order to catch nearly everything, a sample closer to fifteen is recommended. The tradeoff though with that larger number is that your return on investment (ROI) will drop as new usability issues are less likely to be discovered by the latter two thirds of participants.
An important caveat…
What’s often overlooked is that if you decide to test a design with five participants, you shouldn’t just test with one round of five people but work with your team to iterate on the issues you find, then test again with another five people, and so on. The latter studies you will run that come after the first serve as “quality assurance of the outcome of the first study and help provide deeper insights as well”. There should also be “a smaller list of usability problems to fix in a redesign but not all the fixes incorporated in a redesign will work; some deeper issues will be uncovered after cleaning up the interface”.
Having an iterative design process is an important component to the argument of using five users that is essential to it being rigorous and successful. If you don’t apply this process you can be playing with fire if you simply ship a fix to a problem that might be challenging or risky to your product.
“Iterative design is a design methodology based on a cyclic process of prototyping, testing, analyzing, and refining a product or process. Based on the results of testing the most recent iteration of a design, changes and refinements are made.” — Paul Woodward
There are two approaches I’ve used to apply iterative design most commonly:
One approach is to use Rolling Research studies that are scheduled at a regular cadence for your team. Each round brings in a small sample, you test with a functional prototype — then learn, the team applies fixes, and at a regular cadence you then have the next set of people come on, and repeat for multiple rounds until task performance is high.
A second approach is to use Rapid Iteration Testing and Evaluation (RITE). In this approach you again utilize a prototype, but keep a continuous stream of users coming in over several days with brief breaks in between for prototype updates. This approach, while intense on moderation, and requiring fast prototype updates, condenses the process of design and iteration, allowing teams to make rapid leaps forward and to keep testing until all issues have been resolved.
Which approach should you use?
No two situations are the same. Rather than adopt a blanket rule for participant count in your usability studies walk through the following checklist to help weigh your options to choose the optimal participant size and makeup.
- Whats at risk if something is missed?
The risks of poor comprehension of a piece of functionality in some cases may be minimal and carry little risk to a user’s safety, but in other applications these risks could put lives at risk or lead to significant financial consequences for an organization or a user. Do a risk assessment as part of research planning, this will scope your research accordingly so you can all be to consider these consequences and to allow for a larger scope of participants up front or in multiple rounds of testing.
2. Who are we designing for?
Nielsen makes clear that utilizing five participants “only holds for comparable users who will be using the site in fairly similar ways”. You can be certain that most major platforms will have different user groups each with their own needs. Take for example an application being created for University students and Faculty. It will need to be evaluated to ensure that it meets the needs of both groups and possibly administrators. Each of these groups would likely have overlapping as well as some unique task needs in the system. Expand the recruit to be three to five participants from each user group and iterate with these groups as discussed earlier.
3. How much time and budget do you have?
The biggest bridge a lot of stakeholders need to be guided to cross, is to think of usability testing not as a single event, but as a process that requires investment in time and resources until an intended outcome is reached. In my earlier career when I used to QA Software we would go through functional testing while code was early and things were super buggy, but then we would move to regression testing where we would hit the same code round and after round until we got rid of all the major bugs. We need to position good design in a similar way and acknowledge it might take more than one go to get it right.
Good Design requires refinement
Chances are any product you’ve used that felt refined, intuitive, and delightful to use went through multiple iterations to get there.
It’s going to be often challenging to know what the likelihood is of errors being found and to predict the efficacy of solutions as they are developed. It may take one round of testing and iteration to solve a problem, or it might take several. This unpredictability is unfortunately not something that I think our traditional systems of software engineering deal well with.
As practitioners this is an uphill battle, we need to stress the importance of doing design iteration and validation and the importance of thinking of participant numbers as one of a number of considerations when scoping research.
Having often operated in environments where shipping timelines can be tight and incentives are based on launching with deadlines I’ve set plans for iterative testing in advance and proactively recruited participants for multiple rounds of testing often coordinating this and asks for time with Design and Engineering partners. If we need fewer participants we can always reschedule them for later, but if we need them, it’s great to know they will keep funneling in.
If a design is performing poorly and my team is up against a deadline I make it a point to let high level decision makers understand the risks and how it has performed. If the decision maker chooses to move forward with a the design as-is, I ensure that they understand the risks but I also offer to continue to work with my cross-functional counterparts to iterate towards a solution with better usability.
Weighing the risks is something we have to do, and it’s our job as Researchers to “Stop the Line!” when we see risks, but its usually pragmatic approaches such as these that take not only our research and user needs into account but also weigh the needs of the business and utilize iteration effectively that I have seen have great likelihood of success.
