Research That Counts

Get to know quantitative UX research at Facebook

Published in

Meta Research

5 min readNov 14, 2018

In an earlier post, we described How Quantitative UX Research Differs from Data Analytics. Now we’ll focus on what quantitative UX researchers at Facebook do, the kinds of problems we solve, and the tools, methods, and skills we count on.

When and why do we do quantitative UX research?

At Facebook we use a variety of research methods to understand people’s preferences, needs, and pain points. Quantitative UX research helps us answer questions like:

What’s the most common reason people go to Facebook and do a search?
What types of video content are most meaningful to people?
What are the motivations and contexts for people to open the Facebook app?

Quantitative UX researchers collect information by measuring actions, thoughts, or attitudes in different ways, such as conducting voluntary surveys and online polls or analyzing log data. At Facebook, we use structured, measurable data about specific, well-defined constructs or behaviors to test patterns and relationships among variables, or to estimate the likelihood, frequency, or intensity of a social phenomenon — and to generalize results from a large sample population. Our quantitative research reports typically include statistical analyses such as tests of hypotheses, quantitative reports/sizings, experimental design, and ways to quantify and measure attitudinal constructs.

Methods

The most commonly used quantitative methods at Facebook are surveys and behavioral analysis of logged data. We’ll explain each of these below, as well as a few others.

Survey research

We use surveys to gather information on sentiments, attitudes, and evaluations related to different products and features on a large scale. Surveys help us answer why questions and gather insights around motivations and perceptions. Some important, carefully tested surveys later become tracking surveys that are run repeatedly, giving product teams constant access to trends and directions.

Researchers analyze survey data on its own or in combination with other behavioral data such as number of “likes”. We use a variety of statistical approaches on survey data depending on the research questions, ranging from simple descriptive statistics to inferential statistics, general linear regressions, hierarchical and structural modeling, Bayesian statistics, factor analyses, and more.

Log data analysis

While qualitative research helps to better understand perspective of small groups of people, and surveys help us measure the prevalence of different behaviors, log data can uncover patterns of behavior that emerge among the over 2 billion people who use Facebook. Common use cases include:

Connecting perceptions, motivations, and attitudes to behaviors. For example, if we’ve done a survey about a new feature on Facebook Live, we might use data analysis (e.g. of time spent or click count) to identify behaviors associated with high or low satisfaction with the feature.
Testing hypotheses about behaviors. Qualitative interviews and observations often yield findings about behavior patterns. We use logged data analysis to test how well those findings generalize to various markets and segments.
Creating predictive models based on self-reported training data. Survey data can sometimes help us create a training set to create a larger-scale predictive model of how people use Facebook.
Understanding different patterns of use by different groups of Facebook users. We analyze behavioral data such as clicks and time spent by different groups of users (e.g. based on demographics) to explore different patterns of Facebook use.

What skills do Facebook’s quantitative UX researchers typically have?

We have a diverse group of quantitative UX researchers, with various backgrounds and skillsets. Some of us have more expertise in survey methods, while others focus on logged data analysis or a mix of the two methods. But most of us bring knowledge in some or all of the following areas:

Research, first and foremost

The most important part of a quantitative UX research job is to solve people and business problems. To succeed, a researcher needs to know how to scope business needs into research problems, design appropriate plans, and identify valid and reliable methodologies and data samples to answer research questions.

Human-centered data analysis

Our goal is to understand people and improve their experiences with our technology. We take a human-centered approach to research questions, regardless of data size. Many of our quantitative UX researchers have backgrounds in social sciences and HCI.

Deep knowledge of statistics, and sometimes machine learning

In order for us to drive meaning to our product teams, data has to be rigorously analyzed and turned into meaningful insights. Strong knowledge of parametric and non-parametric statistical methods, machine learning algorithms, and some knowledge of network analysis are essential.

Programming skills with statistical tools like R and SPSS, AND database tools like Hive and SQL

Theoretical knowledge of statistics and machine learning is a good foundation, but as quantitative researchers we also spend a lot of time cleaning, processing, analyzing, and visualizing data in tools like R and SPSS. A good understanding of SQL is also helpful, since our data is mainly stored in distributed databases. To get to a target set of features to include in an analysis, we often need to join multiple tables and filter on conditions.

Telling stories with data

Collecting and analyzing data is just part of our work. The most important goal of a UX project is usually to tell a story about a human problem, provide solutions for it, and improve a product based on it. Our data and analysis has to be digestible and compelling to a wide audience of people from different backgrounds. Good visualization skills can help a lot here. We also often enhance the persuasiveness and impact of our research findings by combining quantitative results with nuanced insights and storytelling from qualitative research.

This summary isn’t meant to represent every quantitative UX researcher at Facebook — as we said, we’re a diverse group. It also doesn’t reflect how quantitative research works at other companies. But we hope it gives you some insight into what quantitative UX researchers at Facebook do, how we do it, and why.