"Just talking to 5 users enough!" .. Not Always!
Recently, I have been conducting interviews for UX role in my organization. I have been getting quite troubling answers when I enquire about the research work they did on a project
I ask "how many users did you talk to?" and the number I get is "5 (+/- 2)".
Then, I ask "why?"… here is the answer I get (average paraphrasing)… "You get most input from 5 users"!!
I am less impacted by the simplicity of the answer, but, more by the surety behind the statement without actually understanding the concept. It has happened with far more folks than I expected.
Yes, NN/g has stated that "Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford." But, it is the summary!!! (Full Article)
What are the missing pieces here?
Count of Users
A key paragraph many miss out on is … "When to test with more users". It clearly states -
You need to test additional users when a website has several highly distinct groups of users. The formula only holds for comparable users who will be using the site in fairly similar ways.
So, logically, that magic number "5" is not for total count of users to talk with, rather it is "number of users per group". "Talk to only 5 people and we are done" is most common mistake being made by many (Confession: I have also made so in "very early" stage of career and was corrected by my mentor).
You / Your team are not "The User"
It is easy to ask your team to be the "surrogate" users, but, it is a Ginormous risk to do so. Developers or other stakeholders on the project will always have a default bias towards the product and it will be waste of time to do so.
Yes, you can ask your team-mates and stakeholders to suggest ideas, which will in turn become 'hypotheses' as you go to actual users. Use them to run pilot rounds for your interview protocols, not actual interviews!
Even if you are making an company-internal application, it is critical to talk to users who are not part of your core team working on the app. If the app is for employees to use… their managers are not the "actual" users.
Same principle applies for friends and family as they might have an inherent bias to tell you "good" stuff about the product you are working on.
Iterative Research rounds
Iterative research means that you can run multiple rounds of research with smaller set of users (again ‘user count’ matters). (Mentioned in the same article BTW!) The idea here is to make research rounds quicker and keep an ongoing flow of inputs from research to the stakeholders and dev teams. This resolves any shouts about “UXR being roadblock” from the team.
This method is especially useful for Usability testing for regular iterations of wireframing.
NN/g statement is a guideline… not the final word
Lastly, remember that, based on statistical work done by Nielsen Norman group, they have come to this number "5". But, it is a guideline… not set in stone.
It is a good foundation to have any discussion on how many people we should talk to understand gaps in existing app or for exploring and identifying user needs. So, build your work on top of that.
From interview/portfolio standpoint, work on a valid justification for "why you chose to talk to N number of users?" .. than just saying that summary line!
In case you still need to talk more, just comment here and we can discuss it. If you liked it, please share your appreciation by comment or clap!
