How Quantitative UX Research Differs from Data Analytics
Besides several similarities, there are 4 major distinctions between these two roles.
When we introduce ourselves as quantitative UX researchers, people often get curious about the “quantitative” part of our title. One question they ask is how our work is similar to, or different from, the work of product analysts or product data scientists. We hope this article will provide a comprehensive answer.
Commonalities
The confusion is understandable. For one thing, we both love data. We’re trained to conduct quantitative analyses with variety of different data sources, including experiments, surveys, and logged behaviors. Many of us, across both groups, come from quantitative disciplines such as social psychology, statistics, computer science, and economics, which help us drive insights and elevate our teams’ understanding of product usage through large amounts of data.
At Facebook, the Data Analytics and the UX Research team share the same mission of explaining phenomena we observe in data. We both focus on making meaningful and interpretable inferences about data, relationships between variables, and explanations for changes or patterns in the data. (By contrast, machine learning engineers and those in other big data roles focus on predicting unknowns as accurately as possible.)
Both groups use the same primary tool: statistics. We write code in data analysis software like R, Python, or SPSS. We spend a lot of time exploring and visualizing data to drive our hypothesis generation. We visualize more complex relationships of data points using libraries in R and Python. We also need to have knowledge of distributed data storage systems like Presto and Hive (some SQL knowledge is often sufficient to work with those tools). Most of our work ends up, hopefully, in presentations that clearly and concisely communicate our results.
However, there are also fundamental differences between the two roles. Here are 4 major distinctions between quantitative UX research and data analytics. (Note: these differences are influenced by our work at Facebook and may not all apply everywhere.)
1. Human-Centric vs. Business-Centric
For a technology company to perform well, it has to focus relentlessly on both improving business metrics and delighting its users. Insights about how and why metrics are changing help the company build better products and grow their business value. Understanding users — their motivations, their experiences, and how the product fits into their life — is also critically important.
Quantitative UX research delivers insights about people. UX researchers often approach research projects with questions such as: What are the human motivations for using these products? How do people perceive and use the product? How do they react emotionally and physically to it? What do they like and dislike about specific features? What role does the product play in their daily life?
Data scientists, on the other hand, often start by asking questions related to how the product is performing, or is expected to perform, in the marketplace. How does a product feature change behavioral metrics, such as clicks or time spent? How much adoption of the product did we receive on various devices? Which features are used and which ones are abandoned?
Despite the different motivating problems, data scientists and quantitative UX researchers have similar workflows as they collect and analyze data in order to discover important interactions and relationships between technology and people.
2. User Intent vs. User Action
User actions tell us about what is happening — for example, how many times they clicked on something, how often they come back to the app, or how much time they spent on the site.
User intents, on the other hand, are about the relationship between people who use the product and (un)available product features. For example, out of 10 clicks, how many of those clicks were out of interest and how many were due to frustration? What brings users back to the app and how do they feel about it? How much of the time they spent on the site was time well-spent, and how much was spent looking for something they couldn’t find or trying to figure out a feature that wasn’t intuitive?
Data scientists are less concerned with such questions than with with metrics and collective performance based on user actions or lack of action. They’re interested in the timing, variety, and magnitude of users’ signals — things like views, clickthrough rates, time spent, and churn.
UX researchers, including quantitative ones, are mainly interested in understanding how people use our products, what problems they may have, and what works differently for them. Quantitative researchers seek to gain insights about the intent of people’s product usage through patterns in the data they collected. They also try to measure quality of experience using self-reported data (surveys) or behavioral data.
While data scientists are more concerned about how many people used a new feature and what they did afterward, UX researchers aim to understand how many people used the feature in various contexts, what motivates them to use the feature, and how they felt about the experience.
3. Inference vs. Prediction Accuracy
Like data scientists, quantitative UX researchers may use a multitude of statistical tools to gather insights from data. While the main suite of tools used by the two groups are roughly similar, each group uses the tools differently, since they’re pursuing different goals.
Compared to UX researchers, data scientists are more often motivated to improve the predictive accuracy of their models. (The most accurate models are black box machine learning models, which are hard to interpret by their nature.)
UX researchers are more often motivated by inference. In many cases, we’re not looking to predict future phenomena but to better understand the factors underlying experience or behavior. That’s why UX researchers more often use social science models that are more interpretable but have lower predictive accuracy.
4. Analyzing Survey Data vs. Logged Behavioral Data
In data science, activity logs are the primary source of data. Quantitative UX researchers use a combination of log data and self-reported survey data. Depending on the research questions, we may use only one source of data or combine multiple methods of data collection and analysis.
Analyzing survey data requires a different methodology than analyzing log data. To accurately make sense of survey data, the quantitative researcher must consider, and model, the survey design and data collection process. Survey data is therefore typically analyzed with some form of regression, in which survey design elements are incorporated into the model. For activity log data, Data Scientists typically consider these elements of data collection to be ignorable.
Log data is frequently several orders of magnitude larger than survey data. As a result, overfitting and algorithm speed are critical issues in learning from log data that typically don’t arise in analyzing survey data. While regression models are also used in analyzing log data, Data Scientists frequently use methods involving regularization in order to handle issues introduced by data size.
Working Together
Despite the real differences between data science and quantitative UX research, there are undoubtedly many cases where the two roles are almost interchangeable. But they can also be highly complementary, taking full advantage of a diverse range of backgrounds and skills. In fact, at Facebook, some of the most impactful, satisfying, and fulfilling research projects are collaborations between the two.
The need for both groups shows no signs of fading. The number and variety of meaningful research questions about the relationship between technology and people ensure that data science and quantitative UX research will continue to exist in parallel, driving the business and the technology forward to serve people better.
While we’ve shared some of the ways Quantitative UX Research and Data Analytics are similar or different from each other, our observations are influenced by the context of our organization and the nature of our work. We welcome further discussions about these two roles in other organizations, and fields. What kind of problems do you solve as a Quantitative UX Researcher or a Data Scientist? We’d love to hear from you — please comment below!
Authors: Mary Nguyen, Researcher at Facebook; Saide Bakshi, Researcher at Facebook; and Alex Whitworth, Data Scientist at Facebook (from left to right)
Illustrator: Drew Bardana
