Incorporating Mixed Methods: How big does my sample size need to be for statistical analysis?

This is part 2 of an ongoing series for qualitative research practitioners looking to incorporate quantitative methodology into their work. Part 1 covers When Can I Conduct Quantitative Research? Part 3 discusses Bringing Quantitative Methods To Your Qualitative-Focused Team.

A common myth of quantitative research is that it won’t be helpful if your user base isn’t the same size as Amazon’s or Google’s. Don’t worry, you don’t need millions, or even thousands, of data points to get valid results. All you need is a basic understanding of how a few concepts you’re already using transfer to statistical methods to ensure you interpret your data correctly.

The first concept is that larger sample sizes allow for more precise results. You wouldn’t speak with the same amount of confidence about an insight that you only heard from one person as you would when you heard it from 5 or 6 people. Similarly, unless you survey the entire population to find the true metric you’re looking for, every statistic you measure will have some variability. That variability shrinks the larger your sample is.

To address this, statisticians use confidence intervals, ranges of numbers that it is likely the true number you are trying to measure falls between. For example, if you are measuring the average height of ducks in a city, you might measure 20 ducks and have a 95% confidence interval of 11 to 23 inches, meaning that you are 95% confident the true average height of all ducks in that city is between 11 and 23. If you measure 200 ducks, a 95% confidence interval might be 9 to 12.

Quantitative studies with smaller sample sizes can still be useful and valid, just keep in mind that your results will have larger confidence intervals. There are many online calculators that will allow to see how large your confidence interval will be depending on your sample size; two are linked at the bottom of this story.

Another concept that applies in both qualitative and quantitative research is selection bias, the bias created from sampling that isn’t representative of its intended population. Consider a survey for the U.S. population. If I use a sample of people randomly selected from Facebook, the survey results will be biased, since my sample will not include anyone who isn’t a Facebook user. In statistics, this is sampling bias. Many methods assume your metric was collected from a simple random sample (SRS). A SRS is a sample, or subset, of your entire population collected so every member of the population is equally likely to get selected. However, SRSs rarely happen in real life. Straying from a SRS is the most common cause of sampling bias.

A biased study doesn’t mean you shouldn’t explore quantitative research — there are many times when avoiding sampling bias is close to impossible unless you have unlimited time and money. There are methods statisticians have used to account for bias in their metric. To get started, however, you just need to acknowledge the bias in your study when drawing any conclusions from your numbers. If you give a survey to the top 10% of employees in a company, realize that the results will only be true for the top employees, not every employee in that company.

Qualitative researchers already incorporate these concepts in their work, often without realizing it. For example, there is a high risk in making a design change from something one person said. Therefore, you continue investigating until reaching convergence from other people to reduce the likelihood the one person you spoke to differed drastically from the rest of the group. In addition, you wouldn’t talk to managers and say that you did research with managers and their sales reps; you need to collect data from both groups. Understanding these concepts will help you assess opportunities to incorporate quantitative methods into your practice and interpret the results correctly.

Confidence interval calculator for a proportion: https://www.allto.co.uk/tools/statistic-calculators/confidence-interval-for-proportions-calculator/
Confidence interval calculator with raw data: https://measuringu.com/ci-calc/