A well-thought-out sample (the group of participants you include in your research study) is the first step to curating your data collection. Optimum sampling is all about reducing risk of skewed or inaccurate findings. Skewed findings occur when a result has been led in a direction based on unapplicable entries that leads to overall conclusions based on a few strong, differentiating but unapplicable findings. By making your findings as applicable as you can to the audience you want to learn about by including diverse, relevant, and balanced audiences in your study, you can help ensure that your findings are based on the most relevant perspectives possible.
My background is in sociology, so I was trained on creating samples for social research that eliminate as many facets as possible that could skew the results in one way or another. Sociological research aims to help us understand society better by gathering statistically significant, generalizable insights about culture and behavior. In UX Research, our goal is to paint a picture of the ecosystems our users are part of, how their environments shape their perspectives and needs and help design experiences that address those needs with consideration of the business needs of the company.
UX Research often helps reduce risk for new and improved experiences, including identifying contextual restrictions, and usability hurdles, and roadblocks. It also helps validate if a hypothesized design experience can be built with a high enough level of confidence that the potential negative impact is small enough to not be a critical risk to the business. In sampling, the audience we are trying to learn in UX Research is often smaller and has characteristics that sociologists don’t need to differentiate, like premium vs. basic membership, length of account ownership, or activity levels within a portal.
You can optimize your sample for applicability through a few steps:
Step 1: Who?
First, you have to ask, who is the population you want to make your research findings applicable to? Then, you should establish generalizability (diversity within the sample making the findings more applicable to the population you are studying — demographics, diversity of types of users in that population.) What are the segments within your population? Might there be differences in groups within the population and how they would perceive the interaction/content? These could be free + premium members, high usage + low usage, new users + established, long-time users.
By thinking through how experiences may differ for varying use cases or groups within your population, you can ensure your sample includes an even distribution of people from those groups. These groups often change for every project based on the goals of the experience you’re testing.
There are segments that can apply to every study for a user group, like free or premium members, but also unique ones to the interaction you’re studying. For example, if you are working on a purchasing flow of a dental plan, the experience for someone who is also purchasing on behalf of themselves plus dependents might have a different experience and expectations for the flow than an individual purchaser. First, zoom out to look at segments in the population at large, then zoom into the population that would interact with the specific use case you’re testing.
(Source: https://sociologydictionary.org/generalizability/)
Step 2: How many?
How many people are in the population? Statistical significance and confidence are built by including enough participants to build up statistical confidence, which is different for qualitative usability studies and quantitative studies.
Qualitative:
"The following chart summarizes 83 of Nielsen Norman Group’s recent usability consulting projects. Each dot is one usability study and shows how many users the researchers tested and how many usability findings we reported to the client.” They found that interviewing between 5–10 participants was the richest point of finding many usability issues within a design experience. This is the recommended sample size for qualitative studies to detect the majority of new usability flaws.
Source: https://www.nngroup.com/articles/how-many-test-users/
Quantitative:
When finding how many people you need for a quantitative study, like a survey, it’s important to utilize confidence intervals and margins of error to see how many responses you need. The confidence interval is the level of confidence you can have that your conclusions from the study are correct. For example, the industry standard is a 95% confidence interval and 5% margin of error. This means that you can be 95% confident that the conclusions made from this sample are generalizable to your population of interest.
Here is a link to my favorite sample size calculator:
https://www.surveymonkey.com/mp/sample-size-calculator/
You simply enter the size of the population your experience will apply to, the confidence interval percentage and margin of error you’re comfortable with (industry standard is 95% and 5%, respectively), and it will calculate how many participants you need. So easy!
Step 3: Randomization
Randomization is standard practice in sociological research that can also be used in UX research. Utilizing randomization when possible in UX Research in selecting who to send interview invites to or randomizing questions when it doesn’t interfere with or interrupt the flow of an unmoderated study can help make the study even more sound.
Example strategies:
- Randomly selecting users with varying amounts of experience with the product
- Varying demographic data
- Random selection from the list you are picking participants from
Randomizing the sample can lead to a more evenly distributed population for your study. This also helps prevent sampling large segments from groups within the list since most lists are organized by qualities such as alphabetical order, recent interactions, or time that they signed up to participate. These variants can unconsciously affect the results, and randomizing can help keep a distinct group from dominating a perspective in the findings.
Step 4: Studying extremes
If you enter the lottery, you expect the winner to be truly random and fair. If someone buys many tickets, the odds are more in their favor to win. If someone studies lottery number selections for a living, and then they pick numbers based on their research, they may be more likely to win. Research sampling should fight against any outside influences like these that can affect the results. In a UX Research project, skewing insights might look like a participant that rushes through surveys just for the incentive, providing little to no actual feedback and putting song lyrics in open-ended questions. These kinds of unapplicable responses should be removed.
However, if you want to learn about how people make lottery selections, it might be valuable to interview someone who has never entered the lottery and someone who enters every cycle or an above average amount of times. By studying extremes, you can efficiently discover areas that cover issues that the people in between the two extremes may experience.
An example in UX Research might be someone who is an Instagram influencer and using Instagram to the absolute fullest extent, daily as well as interviewing someone who has never used Instagram. You can learn about the deeper features from the influencer as well as core features and prioritization from the newer user. The influencer is an expert, so whatever problems they have may be impactful to know. Also, understanding the issues of a new user helps relieve pain points that may be hurdles to new users to be able to enjoy Instagram.
It’s best to utilize extremes as part of your sample, not the whole sample, to make sure you get a picture of the overall product experience for real users, like adding extra seasoning to a dish to add depth.
Sampling summary:
It always starts with understanding who the users you want to focus on are, including the characteristics and size of that group. Then, you can calculate how many you need to include and create an evenly distributed selection of participants. Lastly, randomize the participant selection, so you create a more evenly distributed sample.
You can create a rich group of participants by optimizing your research sample through selecting the most applicable participants, reaching the ideal population size, randomizing selection, and considering extreme user groups to decrease risk for your design team and build confidence in the team’s choices, inform prioritization, and increase user empathy among your team members.
Curious to learn more about how social science can be applied to UX Research, their similarities, and differences? Comment and like this article to let us know you want to see more content like this. Thank you for reading!
