Case study: Using diary study to inform user-centered product decisions

TL:DR

This post is a step-by-step breakdown of my process in conducting a diary study to identify use cases for a new product feature.

I benefited from reading other researchers’ experiences and received valuable guidance from my mentors.

However, I found the amount of resources on using diary study as a research tool in the tech industry to be limited. I hope this case study provides tips and inspiration for fellow UXRs who want to leverage more contextual research methods in their practice.

A Desire for Contextual Research

My role is a staff UX researcher on learner experiences at Coursera, an online education platform for career training and lifelong learning. My primary research toolkit consists of qualitative interviews, concept tests, unmoderated user tests, and quantitative surveys. It’s not a complete toolkit. What is missing?

I owe my design research training to Professor Michael Barry and Michelle Jia, whose needfinding course at the Stanford design school opened my eyes to the power of contextual research. One of the first assignments I received was to sit in a cafe and observe people for 3 hours. I went from feeling lost and skeptical in the first hour — to feeling curious and loaded with questions by the time the observation ended.

Before Coursera, I had observed parents with toddlers inside their homes to map out their evening routines, and rode along with food delivery workers to understand how they navigate the complexities of their job.

I realized the missing tool was contextual research. I longed to bring contextual research methodology into the world of digital experiences.

Seizing the Opportunity

In Q1 this year, the right opportunity emerged when my team needed to understand why and how learners would use a new type of learning material.

Normally, we would do concept testing where we show the prototypes to users and ask them to reflect on how they might use them. But this time, I saw the need to go beyond a one-time feedback session in order to identify the use cases learners organically find for themselves in their natural settings.

I proposed to my team that we conduct a diary study over 14 days with high-quality prototypes. This would be the first time a diary study is conducted at Coursera.

Below is a step-by-step breakdown of my diary study process from formulating the research plan to delivering the final presentation.

Step 1: The Plan — Why Diary Study

The first step is to get stakeholder buy-in for doing a diary study. The timeline for a diary study is longer than a typical interview-based qualitative study. So it was critical for me to articulate the benefits of the diary study in relation to our project.

I distilled the benefits into 4 themes:

• “Be there” in the micro-moments that affect decision making — We wanted to identify the triggers for using a certain product feature.
• Observe trends that exist over time, and at frequency — We wanted to build confidence that certain behaviors, especially when it comes to feature usage, repeatable and repeated.
• Go deeper into needs, problems, values, and contextual forces — We recognize Coursera is only a part of our learners’ lives and we need to go beyond their pain points with our product to understand their value systems and contextual forces.
• Minimize the bias of “over-the-shoulder” observation — Again, we wanted to gain confidence that the use cases learners identify for the new learning material we build is natural and repeatable on their own outside the research environment.

For the project team, we had two overarching research objectives:

1. Uncover the use cases learners have for this new learning material
2. Identify triggers and blockers for learning in our learners’ lives

To remove quality issue (as a variable) from the study, we created high-quality prototypes of the learning materials we wanted to test.

Step 2: The Protocol — Minimize the Questions

When I began to design the research protocol, I led a brainstorm session with my project team, including the product manager, designer, data scientist, and learning scientist.

• We first brainstormed the questions we wanted to ask learners right after they completed a learning session. I called these in-situ questions.
• Then we brainstormed all the questions we wanted to ask learners to reflect on at the end of their day. I called these end-of-day questions
• At the end, we thought about “if we could only ask one question, what would it be?” This helped to surface the most critical assumptions we want to validate/invalidate through this research

Equipped with this treasure trove of questions, I initially designed a 20-question daily prompt. After careful — and painful — pruning, I trimmed it down to 10 questions: 5 in-situ and 5 end-of-day questions. On days when a participant did not do any learning, only 2 questions were required.

For in-situ questions:

• Recognizing that participants may not have much time to answer questions on the go, I gave everyone the option to respond in written, audio, or video format
• Example questions — What did you study during this learning session? What activities did you do while you were studying?

For the end-of-day questions:

• I asked participants to share video responses
• I changed the last question everyday to add variety and keep the participants engaged
• Example questions — Could you talk about the highs and lows of your day? If applicable, where does Coursera fit in?

Step 3: The Tool — Whatsapp

The most important 2 criteria I used in the tool selection are:

• Ease of use for the participants
• Ability for participants to share responses in different formats: written, audio, and video

I considered 3 tools: Google sheets, dScout, and WhatsApp. Each had their pros and cons.

I chose Whatsapp because it met both criteria, and provided the best experience for my participants.

Step 4: Three Components of Conducting a Diary Study

For this study, I recruited 12 learners who have expressed an interest in the type of learning material we wanted to test.

Component 1 — Onboarding

No one in my recruitment pool has participated in a diary study before.

For onboarding, I created a lightweight guidebook and walked each participant through it on a zoom call. My goals were to:

• Set appropriate expectations with participants around the level of effort required for the study
• Provide guidance on how to access the prototypes
• Add the participants on Whatsapp and ensure we are able to communicate

I created an analogous example of an exercise diary log to illustrate the level of details expected for the diary study questions, and give an example of a good vs. great contextual photo.

Component 2 — Collecting Diary Entries

Every evening, I responded to the participants’ diary entries. My mentality was to respond as a human first, and as a researcher second. For example, when a participant shared that they lost their laptop, I offered support and told them not to worry about the diary entry for that day.

By staying engaged with the participants throughout the study, I sought to create a positive experience for the participants where:

• They feel that their voices are heard
• When they share something interesting or unexpected, I could immediately follow up with clarifying questions
• When they miss a diary entry, I could send reminders and encouragements

One pleasant surprise was how quickly the participants became comfortable with sharing their personal experiences starting on day 2 or day 3. For example, one participant started showing up in her pajamas and night cap.

Component 3 — Follow-up Interview

Once the 14-day diary study ended, I conducted several follow-up interviews with participants whose diary entries sparked new inquiries.

Step 5: Synthesis

When it comes to synthesis, I heavily relied on the Miro board.

For each of the 12 participants, I created a 14 by 10 grid (excluding labels), where each row represented a day of the 14-day duration, and each column represented a question of the daily 10-question set.

Throughout the 14-day study, I imported all of the participants’ diary entries into the board. I transcribed the video responses onto sticky notes and placed them next to the videos. For written and audio responses, I only included the sticky notes with the written or transcribed messages.

It was critical that I stayed disciplined about uploading the diary entries and transcribing them almost every day. If I had waited till the end of the diary study to process all of the data, I would have “drowned”.

Once I had this rich (and overwhelming) board of data, I knew I needed to be creative about the synthesis process.

• First of all, I wanted my team to build deeper empathy with our users, to a similar level that I’ve been able to through the diary study.
• Second, I needed a system to scale my capability within a short period of time

With these two goals in mind, I organized a series of coding parties with my stakeholders and fellow researchers.

• To prepare for the coding party, I created a library of codes based on my initial review of all the diary entries
• During the coding parties, I assigned each stakeholder to one participant, and asked them to tag that participant’s diary entries with the analysis codes

One wish I had while going through the diary study process was a better synthesis tool. Tagging sticky notes in Miro and then going back to analyze them was not easy. Miro was not built with robust functionalities such as search by tags or grouping insights by tags.

Since this project, I have discovered Reduct, a research tool that would have made my life a lot easier. Reduct combines transcription, coding, and video editing all in one. Recently, they launched a VideoBoard feature, which allows researchers to place snippets of videos onto a limitless board for synthesis — just like how I used Miro board but it would have saved me a lot of time from transcribing entries and importing files.

Step 6: Storytelling for Share-out

I was in a conundrum — possibly the best kind for a researcher. I had way too many insights for one share-out. Under the guiding principle of “less is more”, I explored ways to package the findings in a digestible and engaging format.

Below is the story arc I landed on, after several brainstorm conversations with my teammates, and reviewing my favorite Pixar films (a treasure trove of masterful storytelling):

• Overview of the diary study process — since this was the first diary study conducted at Coursera, I needed to spread the knowledge and advocate for the method
• Overview of the most meaningful sources of diversity within the participants’ approaches to learning — contextualizing learning within our users’ broader lives
• Use cases for the new learning material — answers to the research question
• The story of an extreme user — deepening user empathy and illustrating further opportunities for the team to remove blockers in our users’ lives
• Team reflection and discussion

During the presentation, I added two elements that significantly increased audience engagement:

1. Sharing video reels to bring the audience into the life of an extreme user
2. Using Slido interaction slide to show live polling results and word clouds

At the end of the day, the best measure of success for a research project is in its Impact:

💙 Hearts: My stakeholders and the audience built deeper empathy with our learners

🧠 Minds: Using the reflection question “I used to think… Now I think…”, I was able to read the various ways in which my teammates’ perceptions and mindsets towards the problem space and our users have shifted

💪 Actions:

• The project team began developing the MVP with high confidence
• The insights spawned many new ideas for the product roadmap and future research
• My fellow researchers were inspired to conduct diaries studies
• The social media team built upon my extreme user story for an inspiration post on the company Instagram (with the learner’s permission)

Reflections

The power of contextual research is unparalleled for changing not only minds, but also hearts.

The effort required to run a diary study is double that of an interview-based study. However, the benefits are more than worth it.

I am deeply grateful for this opportunity to conduct the first diary study at Coursera to inform user-centered product decisions, and expand our research toolkit.

🙏 Many Thanks to:

My mentors: Michael Barry, Michelle Jia, David Schrimp

Coursera crew: Polina Varela, Nikkie Ella, Sandra Betancourth, Jonathan Stone, Rick Bond, Ron Delaney, Sedinam Worlanyo

and other researchers:

How to Conduct a Diary Study: A Start-to-Finish Guide by dScout

How to Create a Diary Study (Plus Example)

Using Digital Diaries as a Research Method for Capturing Practices in Situ

Diary Studies: Understanding Long-Term User Behavior and Experiences