How to Deliver Value and Control Research Costs
By Tom Mackey, UX Researcher @ Chewy
When planning research initiatives with stakeholders, one of the biggest considerations is sample size. How many participants do we need to recruit to collect high-quality and reliable data? And how can we do high-quality research while managing our costs? In other words, how can we manage sample sizes in order to optimize our value as researchers? After all, it is our responsibility to help achieve business objectives through a scientifically sound research approach.
How can we achieve optimal value when it costs resources, people, and time to carry out high-quality research?
The Costs of Large, Generative Studies
In every study, there’s an opportunity to increase revenue and decrease expenses. Lower research costs can help establish buy-in from key stakeholders. This is especially useful for large-scale, generative studies that demand statistically significant sample sizes and in-depth feedback.
Depending on the study, sometimes users can be difficult to recruit. Whether these recruitment efforts are handled by a third party or in-house, they can be costly as a result of contracts with vendors, compensation for participants, time, and effort. With all of these costs to consider, we need to strategically manage sample sizes to deliver value. This simple but effective approach is tremendously helpful because:
1. Smaller sample sizes can reduce recruiting costs.
Recruitment can be one of the most expensive components of any study. By opening a study with a smaller sample size (five to seven participants) and recruiting in similarly sized “waves,” you can avoid getting potentially unnecessary additional feedback.
Here at Chewy, we apply the fundamental UX practices of rapid iteration and the golden number of five when we’re evaluating a new opportunity or feature with users. We recruit in waves of five participants (sometimes seven, in case of no-shows/poor-quality participants) using an in-house database of professional users.
2. The costs of recruitment can go beyond just upfront costs.
Recruitment can be an exhausting and laborious process, especially in cases with specific or complex target demographics. Additionally, recruitment has implications for public relations. Not every user is happy to get bombarded with solicitation emails or calls. They may not have the time or desire to participate in a user study.
At Chewy, we don’t recruit more participants than we need and make the most of our time with these participants. By speaking to participants in waves, we can evaluate the results of the study and formulate new questions, omit questions that aren’t as effective, or skip questions that we received sufficient feedback on.
3. Moderation, analysis, and delivery takes time.
Simply put, it takes a lot less time to moderate and analyze five, 45-minute-long sessions at a time than it does 20.
These practices help us conduct user-centered studies, in which user feedback dictates which direction we take studies in. Users help inform our understanding of the problem we are seeking to solve and our approach. By minimizing the liabilities of these large research efforts, we are able to deliver the most value and gain buy-in from stakeholders.
by Tom Mackey
UX Researcher @ Chewy
If you have any questions about careers at Chewy, please visit https://www.chewy.com/jobs
