Why user researchers should care about sample size
User researchers often find themselves having to defend small sample sizes in their research.
Some kinds of research really do require large sample sizes and it’s common for researchers to have to explain why that doesn’t apply to user testing or research interviews.
Here’s the problem: constantly defending small sample sizes can make us complacent. We feel that user research is a sample-size-worry-free zone, but it isn’t.
What’s the deal with sample size?
I was always told that you need around 6 to 10 participants for user testing. Nielsen has a good article about why you only need 5.
You only need a small number of participants because the objective is to uncover potential issues with the design being tested. Nielsen has shown that 5 users will uncover about 85% of those issues.
You can apply the same logic to research interviews as well. If you use them to explore the various experiences and opinions that are out there about a specific subject, you will uncover most of them with a small sample. However, in my experience, interviews are usually broader in scope than user tests, so you would probably need more participants to get to 85%.
So why do we sometimes need large sample sizes?
Perhaps you don’t want to just uncover problems or insights. You might want to know how common something is.
Let’s say I want to know how many of my product’s users are women. If I speak to just 5 of them and 4 of them are women, I couldn’t confidently say that 80% of our users are women.
So instead, it would be better to survey a few hundred of them, which would give me a more accurate percentage. I can then use statistics to work out what the error margins are, which will tell me how confident I can be that the true percentage falls within a certain range.
So what’s the problem?
There are two problems.
Firstly, some people think that you always need a large sample size for any research to be valid. This can make life difficult for user researchers who have to convince them otherwise.
Secondly, by digging in on the side of low sample sizes, we (the researchers) can end up missing the examples in our own work where we needed a large sample size but didn’t have one.
When are we getting our sample sizes wrong?
If you think that sample size and statistics isn’t an issue in user testing or research interviews, there’s a chance that you might fall into one of these traps.
1. Comparing two versions of something
Let’s say you have two versions of a particular webpage and you want to know which one is better. They’re prototypes, so you can’t just do an A/B test.
If you do user testing with 5 participants, you won’t be able to have much confidence that the version that performs better would still be better if you tested it with everybody.
Instead of treating user testing like it’s an A/B test, researchers should stick to the process of identifying problems, making changes and then retesting. It’s better to focus on creating the best possible option, rather than selecting what appears to be the best performing option.
2. Percentages and proportions
If you have a large enough sample size you get to say things like “63.1% (±4.6%) of our users are women.”
If you do some user testing or interviews with 6 participants, you can calculate a percentage like that, but it wouldn’t be very precise or helpful.
Even if you don’t use a percentage, you might be tempted to say that “most” users do something. That sounds safer, but it’s just another way of saying “more than 50%”. You’d still need a large sample to be confident about that.
The same is true of negative claims like “our users have no difficulty doing X”. User testing can tell you that some people have a particular experience, because you’ve seen it happen. But it can’t tell you that nobody has a particular experience. That’s the same as saying “0%”.
3. Suggestive language
Even if you don’t make claims about users in general, you can still cause trouble.
If you report that “4 out of 6 participants were able to complete the task”, you’re making a factual claim.
But unfortunately there’s still a problem. Maybe you aren’t generalising about all of your users, but you’re handing a loaded gun to whoever reads the report.
That kind of statement encourages the reader to think in terms of numbers and proportions, which can very easily lead to generalisations. It also encourages them to rank the findings in terms of how many participants had each experience. Again, this is the sort of thing that requires a large sample size.
We all need to care about sample sizes
I decided to write this post after listening to an episode of the Mixed Methods podcast, where Aryel Cianflone talked to Matt Gallivan about how researchers feel under pressure to be scientific and more like data scientists.
Matt talks about facing skepticism about small-sample-size user research at Facebook and how he has dealt with that issue. They both make a case for embracing uncertainty in research, which is something I definitely agree with. It’s an interesting episode and a great podcast in general.
But I think that they create a bit of a false dichotomy, with user researchers and uncertainty on one side and data scientists and error margins on the other. I don’t think we can do our jobs well unless we understand some of that science too, if only to know what we can and can’t do with our data.
An understanding of sample sizes affects research design, analysis and reporting:
- Choosing the right method with the right sample size to match the questions we’re trying to answer.
- Understanding what we can and cannot conclude from our data.
- Communicating our findings in a way that makes it clear what is and is not supported by the sample size.
We don’t have to become data scientists, but we do need to understand statistics well enough to avoid making mistakes. Or at least, we need to make sure that we are working very closely with people who understand statistics, so that we can get their input when we need it.
Lyst is looking for another user researcher to join the team. So if you like statistically sound user research, click on that link and send us an application.
We’re also hiring for a number of other roles too, including designers, product managers and developers.
