Conducting Diary Studies via Messenger

Key learnings from conducting my very first diary study with Messenger

I recently conducted a diary study via Messenger. (If you’re not familiar, a diary study is a qualitative research method used to collect data over a period of time. Participants keep a diary to track specific behaviors of interest.) Not only was this my first time conducting a diary study, but it was also my first time using Messenger as a research tool. I learned a few things along the way, and I thought I’d share some key takeaways.

Why Messenger?

First of all, I am a heavy Messenger user. I use it every day to talk with coworkers and friends. As you can imagine, at this point, I am very comfortable with this tool and its capabilities. This familiarity certainly contributed to my decision to try it out as a research tool, but there were a few other things about using Messenger that made it seem like a good fit for this study:

• Keeping in touch on the go. I wanted to ping participants three times per day (morning, afternoon, evening), asking a few simple questions about feature usage. Messenger was an easy and lightweight way to do it. I could easily communicate with participants even when I was traveling for a separate research study and sitting in the taxi from the airport. Not being tied to a laptop is a huge plus, especially if your study spans a Saturday or Sunday.

• Getting fresh, accurate insights. If we had asked these questions at the end of the day, we would have lost the data around usage throughout the day, and we’d likely get inaccuracies in data due to problems with participant recall. In the absence of a setup that lets you ask questions in-product directly after feature usage (which would be awesome!), this might be the next best thing.
• Following up on the fly. When participants give you shorter or less descriptive responses than you wanted, you can follow up to encourage them to dig a bit deeper for richer answers. For example, at the end of the day, when I asked participants to submit a screenshot of a post they didn’t want to see, and to explain why, sometimes they’d respond with something simple like, “I didn’t like it”. I often needed to follow up to understand exactly why they didn’t like it. I knew I wanted to have fluid conversations with participants; Messenger allows you to do this.
• Working with a small sample. I recommend using Messenger for smaller sized studies. Because of user privacy, you’ll need to message individuals one by one — and since you can’t schedule messages in advance, you’ll need to be “on call” when they respond. In this particular example, I had 6 participants in my study.
• Collecting rich media. You’re able to send lots of different types of media via Messenger — text, of course, but also photos and videos. I knew that I wanted to ask participants to share screenshots of their in-app experiences; Messenger makes it easy for users to send images and other media types.

Before day 1

Okay, you’ve decided to use Messenger for your diary study. Here’s how to prepare for your study:

• Decide how often to ping participants. I picked three times a day because I was interested in how their usage changed throughout the day. (This was a commonsense decision, not a scientific one.) Think about the right cadence for your particular study.

• Be sure you have the bandwidth. You might need to clear your schedule a little bit to make sure you’re available at scheduled times to ping participants across different timezones and to follow up promptly when appropriate. If you’ve got the questions predetermined (and you should), it doesn’t take long to copy/paste in Messenger, but you need to be available.
• Prep your participants. Talk to participants over the phone before the study begins to clarify the format of the study and set expectations. For example, in this study, I explained that they’d get pinged three times per day, it would take roughly 20 minutes total per day to respond to the questions, and they were expected to respond in a timely manner to each ping.
• Send a test message. While you’re on the call, send each participant a message on Messenger and confirm that they got it.

During the study

Now that you’ve done the prep work, the process should go pretty smoothly. But here are a few ways to maximize your takeaways.

• Expect delays. Some participants will respond right away to your Messenger messages; others might take hours to respond. Even if you set clear expectations before they agreed to participate, be prepared for delays. Be ready and willing to adapt your plan as necessary for different participants.
• Follow up fast. Be sure to take advantage of the ability to follow up and probe with additional questions. The sooner you do it, the more likely you are to still have the participant’s attention. It can help to prepare a potential follow-up to predictable responses. For example, if the person says they haven’t used the feature since your last message, you might ask if they’ve used a similar tool.
• Organize incoming data. Once responses start coming in via Messenger, you’ll want to port your data elsewhere, outside of Messenger. I used a simple spreadsheet to store participant responses as they came in — both text and images. I used a separate tab for each day, one question per row, and one participant per column. This sounds simple (and it is!) but it’ll make your life easier when it comes to sifting through responses for analysis. I could easily scan responses for a given question across participants.
• Show stakeholders. Invite key stakeholders — in my case, a Product Manager, Designer and an Engineer — to follow along as you collect data. If something seems noteworthy (good or bad), point it out to your stakeholders so they can act immediately. Keeping stakeholders informed is a standard best practice, of course, but its important enough that it warrants a reminder every now and then. I’ve found that if you loop in stakeholders early, they’re more likely to feel invested, and ultimately, make decisions based on the research findings.

After your study, when you’re presenting your findings to the broader team, talk a little bit about what a diary study is and show them what one looks like as part of your report. (This isn’t specific to studies conducted with Messenger, but your product team might be wondering what the heck a diary study is!) For example, I decided to show a snapshot of a single day in the diary study with participants’ responses verbatim. Pulling out key quotes from participants is also a great way to make your deck come alive.

Example of data collection.

Using Messenger for a study might feel a little strange the first time, but I think you’ll find that with a little practice, it feels like a natural tool for the job (assuming the circumstances are right, as noted above). If you’ve used Messenger to conduct studies — or if this post convinces you to give it a shot — I’d love to hear your own tips and best practices.

Author: Lauren Vilders, Researcher at Facebook

Illustrator: Sarah Lawrence