Five is Still the Magic Number when it Comes to Usability Testing
Like many things in life, usability testing comes down to quality over quantity.
Here at Prototype1, people occasionally ask us how many people are required for prototype usability testing. The more complex answer to this question is, “it depends.” The basic answer is five. Yes, just five. Our clients are often shocked to learn that we suggest so few users, but we have a pretty solid foundation for why we’ve chosen that number. Five users offer enough diversity in user behavior when it comes web usability analysis, and the cost-return benefit certainly sweetens the deal.
Extensive testing can drain your time and resources, with high costs that accrue when aggregating testing participants who must be screened, scheduled and incentivized. The goal is to improve your site or app’s design by identifying fixable problems and not just documenting design weakness and — often most important — improve the design without draining the project’s budget. This method works best on new products or brand-new features, not things that have been live for a while.
Why We Jive with Five (Users)
The five-user method is hardly anything new in the world of web usability testing. It took storm back in 1993, when Nielsen Norman Group Founder Jakob Nielson researched the strategy and presented his findings. “Elaborate usability tests are a waste of resources,” Nielson claimed. And he seems to be right 23 years later, with Usability.gov still backing his study. Naturally, your test group should be carefully selected to represent your target audience, as you don’t want to be challenged on amateur issues.
Neilson found that after testing with just five users, up to 85 percent of the usability problems already became apparent, so you can run other tests to address the rest. “Small-batch” UX tests are actually proven to be more valuable than tests with multiple users, primarily because adding more users only reiterates the same problems that you can identify with only five subjects, yet typically costs exponentially more. You can use the same five users to test in various rounds.
• Round one: Testing with just one user already uncovers up to a third of the design’s usability. You’ll learn big-picture problems in this round.
• Round two: In the second round, you’ll be able to identify problems associated with QA, information architecture, task flow and deeper user needs.
• Round three: The third round of testing will delve into deeper design issues and help you identify inconsistent glitches that don’t occur all the time.
When Five Isn’t the Magic Number
There are still some scenarios where you need to test for more users. When your user base varies — i.e. if you’re trying to target, say, men and women or parents and kids — you should test more users. if you want quantitative results and your goal is to gather statistics, at least 20 users should be tested. With card-sorting or eye-tracking studies, you’ll need more participants still, at least 15 and 39 users respectively.
We also believe that if you aren’t seeing any glaring patterns in your testing, it’s worth investing in tests targeting another set of users or revisiting the prototype to identify other inconsistencies and issues. Simply put, web usability tests should uncover problems. If it isn’t calling out any weak spots or glitches, it’s probably not doing its job. With the right strategy and a well-chosen test base, though, you should always be able to find ways to improve. After a product is launched, Prototype1 uses cool usability tools like hotjar to aggregate larger-batch analytics on visitor behavior and to find issues that aren’t on our radar.
Unlike buying a new car where taking more test drives is almost always better, app and website testing often gives you more for less. In some scenarios, you can eliminate extensive testing in favor of small-batch tests that garner the same, and sometimes better, result. Stay tuned into the Protoype1 blog to learn more about the five-user method and how you can effectively convince management to get on board.
Originally published at prototype1.io on April 22, 2016.
