The Numbers Game: Humanizing Data in User Research
Using quantitative user research to complement customer insights
Most people associate user research with more qualitative methods, like interviews for usability testing or diary studies. These methods are generally used to obtain insights into things like customer needs, product usability issues, and customer journeys. A common method is focusing on observable behavior by actually watching and interviewing customers while they use a product or prototype.
Observable behavior is a fundamental aspect of user research, for what users say often differs from what they actually do. The emphasis on a user’s behavior helps the researcher to focus on asking the “why” questions: Why do users behave in a certain way? — Is it because they do not understand how something works? Is it because they do not want to do it? Is it because they expect something different? Is it because it does not solve a problem they are facing? To answer these questions, we keep the abilities, motivations, and experiences of the individual constantly in mind. For example: A person motivated to search for a specific product will behave differently on the website than a person searching for inspiration.
Yet, while these qualitative methods give us invaluable insights, there are even greater possibilities when we combine the qualitative approach with quantitative methods and thinking.
But what exactly does such quantitative user research entail, and how can it help us to understand more of our customer’s behavior?
Diving deeper into quantitative user research
I’d like to start with a recent project example in which our user research team combined qualitative and quantitative methodologies. We wanted to better understand how our users experience the editorial section of the Fashion Store, and how we could make the section more inspiring and engaging. We began by inviting users to our in-house research lab, observing them make use of the editorial section and subsections, and asked questions regarding their expectations, difficulties, and overall understanding of the content. We anticipated that people would have varied reactions to our site, based on how fashionable people considered themselves. We therefore recruited participants based on diversity in fashion competence.
Based on previous qualitative interviews, we had new research questions and hypotheses that we wanted to explore and test. For example, we guessed that users with a high fashion competence would use other sources for fashion inspiration compared to users with a low fashion competence. To test this hypothesis we used a quantitative survey.
The survey provided another measure of fashion competence by looking at our users’ understanding of the editorial section and their awareness of inspirational content on Zalando. This helped us not only in validating some of the findings from our interviews, but also in gaining additional insights into how and to what extent fashion competence influences the perception and consumption of inspirational content.
In the future, we definitely hope to integrate more on-site observation data into our research. With surveys behavioral data is, by definition, not very precise as it is based on the self-reported memory of the participants. However, pairing survey data on emotions/motivations/intents/personality traits that might influence behavior with actual on-site behavior data can generate completely new knowledge in areas like developing personalization features or tailoring them for shopping intentions.
What does this tell us?
One learning that resulted from the above research is that quantitative user research uses the same mindset as qualitative user research: focus on behavior (how did they use what) while keeping individual traits in mind (e.g., their fashion competence, emotions, motivations etc.). Quantitative user research doesn’t merely look at “how much?”, but can also offer answers to “why?”. This is best accomplished by combining data on behavior with insights on the person. However, in quantitative user research we do this with standardized measures and in bigger samples. Here, we are not merely observing three or five participants, but rather 50 participants or more. In the end these results merits analyses that use descriptive as well as inductive statistics, enabling us to reliably test hypotheses that were generated through the qualitative research.
Quantitative user research doesn’t merely look at “how much?”, but can also offer answers to “why?”.
Other research methods commonly used for quantitative user research are associated with remote testing (e.g., card sorting and unmoderated task based tests). For example, small scale A/B tests (comparisons between different variants) make it possible to compare customer impressions of different product versions before they are finalized. While such tests are not as reliable as a classic A/B test when it comes to statistical power (i.e., how likely it is to discover a significant effect), sample size, and measuring of KPIs like Conversion Rate, they can be a low-key solution for comparing different mock-ups or prototypes early on. Furthermore, existing solutions that are not yet live in every country can be shown to customers before bringing them live.
Remote testing studies also offer the possibility of asking customers about their experience, granting additional insights on why one they believe one version performed better than an other. We recently conducted such comparisons with our French customers, researching their perception of different sizing help features (i.e., size chat, size recommendation, etc.) that were not yet live in France. This helped us to establish a ranking of which features were considered most helpful
Final thoughts
Insights from quantitative user research are used to complement the findings we’ve established with our qualitative research, sometimes by having numbers confirm observed phenomena, and sometimes by enabling comparisons between customers or products. Quantitative research can also build new bridges with market research or A/B testing by speaking a more similar language in ours and their results. However, there are also risks involved: Getting lost in numbers instead of really listening to what the customer says while using the product.
Quantitative research builds new bridges .
The user research team at Zalando has only just started to uncover all the possibilities that quantitative user research offers. We are excited about the road ahead. If you have any questions about our thoughts and processes, or would like to contribute your own ideas regarding quantitative user research, feel free to get in contact via email at franziska.susanne.roth@zalando.de. I would love to hear from you.
Dr. Franziska Roth is a user researcher with Zalando.
