Quant Diary Studies: An Introduction

Quantitative diary methods can provide a rich, detailed look into people’s everyday product experiences over time. Here’s how to decide whether and when to use them.

One of the most exciting parts of working as a UX researcher is investigating big, complex questions: Why do people use our products? What perceptions, emotions, and attitudes predict whether they’ll use a product day over day? How can we make people’s experiences better?

Quantitative diary studies are one way to tackle questions like these. Here’s a look at when, why, and how a quantitative diary study can bring unique value to user research.

What is a quant diary study?

In a quantitative diary study, participants respond to a short survey every day for an extended period of time. These studies, which are also known as daily diary studies, experience sampling studies, and, most generally, intensive longitudinal methods, are used in several fields of academic research, including social, developmental, and clinical psychology. In fact, both Katherine (my co-author) and I used these kinds of methods extensively in our academic work before joining Meta.

Quant diaries differ markedly from other commonly used UX research methods. One-time cross-sectional surveys allow researchers to study experience at scale, but provide only a general snapshot of people’s experiences and don’t easily reveal day-to-day fluctuations. Qualitative diary studies allow for depth but are less useful for quantifying patterns at scale or drawing numeric comparisons about things like the strength of daily retention drivers and specific product experiences.

What can a quant diary study reveal?

1. How people act “in the real world.” Quant diary studies capture true-to-life experiences by asking people about their perceptions and feelings on a particular day, in their natural environment. This can provide a rich glimpse into what’s happening in people’s everyday lives, with granularity and vividness that sometimes gets washed over in one-shot surveys. Qualitative daily diaries share this advantage but don’t help us do things like numerically comparing groups of people or experiences or drawing generalizable conclusions at scale.

2. How the same individuals change over time. A single noteworthy experience, or the accumulation of smaller experiences over time, can contribute to falling in love or not with a product. Quant diary studies follow the same individuals over multiple days, allowing researchers to capture both days on which certain behaviors or experiences take place and those on which they do not. This opens the door to especially powerful conclusions because we can essentially compare people to themselves.

3. How people differ. Different people don’t respond to a product or experience the same way. In the context of online dating apps, for instance, people tend to receive different amounts of likes — and might also differ in the degree to which receiving likes motivates them to continue using the app. Quant diary studies afford a unique opportunity to gauge how — and begin to examine why — people differ in their change processes as a function of individual and contextual characteristics, such as gender, age, or prior experience. This can lead us to discover, for example, that getting more likes might get men to use the app more, while having no effect on how much women use the app.

A fictional quant diary study to drive product impact

To illustrate the value of quant diary studies for shaping product recommendations, we’ll create a fictitious example based on our work as researchers on the Facebook Dating team. Let’s imagine that the team is interested in knowing whether browsing more (vs. fewer) dating profiles predicts how satisfied people are with Facebook Dating over time. The team’s hypothesis might be something like: When people browse more (vs. fewer) profiles, they will report higher satisfaction with the product.

In a quant diary study, we might test this hypothesis by having people respond, each night for three weeks, to a quick survey that asks how many profiles they browsed on Facebook Dating that day and how they feel about the product.

Our results might look something like this:

The solid lines show the (again, fictional) effects for individuals, with each color representing the data for one person. The dashed line shows the trend line for these effects, cutting across all individuals. We’ll first review the hypothetical insights, and then dive into how a researcher might draw actionable conclusions and recommendations.

Key insight #1: The typical person is more satisfied with Dating on days when they browse more profiles. This “within-person” effect, represented by the solid lines, suggests that giving people a few more profiles to browse each day might lift their daily product sentiment. This could lead us to offer recommendations such as:

• Notify people when they have new profiles to browse.
• Nudge people to browse more profiles through design choices (e.g., adding an “express” or “fast track” button to help them quickly navigate to the profiles surface).
• Prompt people to expand their filter settings to reveal additional profiles.

Key insight #2: People who tend to browse more profiles overall are less satisfied. The dashed red line shows that people who tend to browse more profiles on average — across the entire diary study period — tend to be less satisfied. This “between-person” effect could signal an efficiency problem; perhaps these people are having a hard time finding profiles they like or successfully matching with people, and therefore must browse many more profiles to achieve the outcomes they want. This might lead us to make recommendations such as:

• Ensure that high-volume browsers see the most relevant profiles early in a browsing session. This could help them get matches quicker, without having to go through so many profiles.
• Conduct more research to better understand the experience of high-volume browsers, including their levels of satisfaction with the profiles they see and/or whether their interest is less likely to be reciprocated.

Key insight #3: Individuals differ in how profile browsing relates to satisfaction. The different angles of the solid lines show that people differ in the extent to which browsing more profiles on a particular day relates to higher daily satisfaction. Compare the steep upward slopes of some lines, for example, with the horizontal black line. Uncovering and quantifying how people differ in their effects is a noteworthy strength of quant diary methods. In this example, such a result would tell us that finding ways to boost profile browsing might be more important for particular groups than for others. It might also lead us to offer recommendations such as:

• Partner with data science to define user segments or profiles.
• Assess whether people showing greater boosts to satisfaction are more vs. less likely to continue using the product over time. This would help suggest potential longer-term implications of product nudges to boost profile browsing.

As these examples show, quant diaries can open the door to new insights and recommendations that would be difficult, if not impossible, using cross-sectional survey methods. We’d also miss important opportunities to improve the user experience. And although our example is hypothetical, the basic advantages and considerations are relevant for a wide range of products.

How to decide whether to run a quant diary study

Now that you’ve had a taste of the utility of quant diaries, here are some guiding questions to help you decide whether and when to use this method.

• Is my research question a good fit? Because quant diary studies are fundamentally about capturing how people change, they’re best suited to research questions about a change process. Before committing to a quant diary, you might consider conducting pilot research or engaging a cross-functional partner, like data science, to get a sense of variations in daily behaviors and experiences. If you don’t anticipate that behaviors will change much, a one-time survey or interview might be perfectly sufficient.
• Do I have the resources? Quant diaries are resource-intensive. Participants need to be compensated for several days (or even weeks). They also take time to run. A quant diary study might last anywhere from a week to longer than a month, which can mean building in more time for data collection. If you’re working with a vendor, administration and management to prevent attrition can get expensive.
• Do I know enough about my product space? Make sure you know enough about your product space to draft a short, focused survey. Because people will be completing the same survey every day, it needs to be concise to prevent attrition. To get more familiar with the product space and narrow in on the highest-priority questions to include, it can be useful to conduct foundational research first, using a traditional survey or interview approach. In our Dating example, before embarking on a quant diary study, we’d want to see preliminary signals from earlier research and analysis about the importance of profiles.
• Do I have the statistical expertise to analyze data from a quant diary? Quant diary studies typically require an approach known as multilevel modeling (aka hierarchical linear modeling or random effects modeling) to account for the fact that multiple responses will be collected from the same participants over time. If you’re not familiar with multilevel modeling, you might want to build up your knowledge before attempting a quant diary, or to engage a collaborator or vendor partner with expertise in this area.

We’re excited about delivering the impactful insights that quant diary studies can uncover, and we hope you’ll consider one for your next research project.

Author: Catalina Torrente, UX Researcher at Meta

Contributors: Katherine Zee, Former UX Researcher at Meta, and David Kille, Research Manager at Meta

Illustrator: Drew Bardana

Learn more

For more on quant diary methods, including how to implement them and analyze the data, and what exciting inferences can be drawn, here are a few resources:

• Diary Methods: Capturing Life As It Is Lived: Academic article charting the use and benefits of quant diary methods within psychology.
• Intensive Longitudinal Methods: Book discussing the analysis of quant diary data, with accompanying website that provides example datasets and code for R, SPSS, and SAS.
• QuantDev Analysis of Experience Sampling and EMA Data: Website with tutorials on analyzing quant diary data.
• Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence: Book describing multilevel modeling, from foundational concepts to advanced data analysis.