Lessons from Running A Diary Study
In the sea of endless in-depth interviews and usability tests, here came the most anticipated research method in my career as a researcher — Diary Study.
Diary study was a big deal to me, given the logistics and the people management to make it happen. It was a mix of excitement and a tiny spike of anxiety, alas I have a dream team in Somia CX who always made things happen.
The scale of the project was quite big (at least to me). In this study, our team had to manage 30 respondents for a 5-day diary study. A task will be assigned to the participants every single day for 5 days in a row. Given the assignment, it was a little bit worrisome as the participants might bail out at times.
What is a diary study?
A diary study is a series of research that spans for couple of days, weeks, or even longer. In the user research context, usually during a diary study, a participant will be required to do certain tasks and record them daily. The task can be recording a video, answering survey questions, or keeping track of certain behaviors.
In our case, the participants were asked to fill in a survey and upload screenshots of a social media app. The diary study submissions were used as probing materials during the IDI sessions. We collected the data one by one and created a stimuli deck personalized for each participant. It was handy because the visual aid helped them share the contexts and personal stories.
Things to watch out for: duration vs dropouts
Given the duration of the study and the task given, it was a bit daunting. A longer study equals extra backup participants as the dropout rate might be high. That’s why, if you need to collect 20 data, you might need to recruit 10 more participants as backups.
The longer the duration, the trickier it becomes. In user interviews or usability testing, if the participant bails out, you can easily find a new replacement and schedule a new session. However, for a diary study, it means you are back to square zero. The research duration will last longer as you need to onboard the participants and they have to start again from Day 1 while the rest of the participants are already further in the diary.
How did we manage a diary study in Somia CX?
Our diary study was conducted in December 2023. The dream team that made this project happen consisted of Kara, me, Daniel — our Research Ops, and our former intern, Aaya.
The project had two groups of participants, 1 group of parents and 1 group of teens. For each group, we had 15 participants. Each researcher was in charge of managing comms with 10 participants, mixed of the parents and the teens.
The diary study was 5 days long and all participants must submit their tasks every day. The task itself took 15–20 mins to complete. They were asked to browse their social media as usual, take screenshots of the posts they saw, and share with us why they picked the post.
Since we had to ensure that all 30 of them could finish the complete round of the diary study, our team came up with these strategies.
1. Facilitate a briefing before the study starts
The project timeline was tight. It was December. In the school calendar, early December is an exam week for the students. We were racing with the timeline and the participants’ as well.
That’s why, ensuring participants’ understanding of the study and their task was important. We hosted a briefing session with a Q&A. We showed them how to upload and submit the task.
We couldn’t afford to make an assumption that every participant was able to understand the tasks and navigate around the diary platform on their own. We were glad that we hosted the session, turned out a few participants were still unclear about the study and a bit confused. The briefing helped us ensure that we as researchers and all of our participants were on the same page.
2. Create a personalized WhatsApp group with each participant
For 5 days, all three of us were glued to our WhatsApp apps to run and monitor the study. To keep track of the study and to keep in touch with the participants during the diary, we created a WhatsApp chat group for each participant. If we were to pool all 15 parents and 15 students in one group, we might not be able to keep track of the amount of chats sent in. There would be too much “noise”.
In each group, one participant will be accompanied by one researcher. Creating a personalized WhatsApp chat group with each participant indeed was a lot of work to handle but pooling 30 participants into 2 groups would be quite tricky. With a personalized chat group, we were able to monitor the submissions and ask follow-up questions if needed.
Although we started the project with a deep sigh as all three of us each will manage 10 chat groups, the personalized group system was a lifesaver. Some of our participants’ submissions didn’t meet the criteria of the instructions hence they had to resubmit. In some cases, the participants didn’t write a description and we had to ask them follow-up questions to clarify.
3. Set up daily deadlines, but be reasonable
Another way to keep the diary manageable was setting up ground rules for the participants. We would share the daily tasks at the same time of the day throughout the diary study. As for them, the deadline was also set.
Besides keeping track of the submissions, and checking the contents submitted, we also had to manage the Indonesian to English translation of the submissions. Hence, setting up the same deadline was helpful since we could manage our time better.
Although we have set a daily submission deadline, we were open to negotiation. Our diary study timeline coincided with the students’ exam week. We understood that free time in an exam week was indeed a luxury. Some of them reached out to us asking for more time nearing the deadline.
Things I’d do differently for diary study
After wrapping up the project in early 2024, I was assigned to another project that involved homework submission as a pre-interview round at the end of 2024. Similar to the previous diary study, the participants were also tasked to fill in a survey and submit screenshots of certain apps.
So, after doing one round of diary study and now jumping to another similar one, here are things I’ve done differently.
1. Going back to good ol’ Google Form
Instead of using a fancy platform for data submission, I utilized Google Forms to collect pre-interview data. With participants in mind, I tried to create a concise list of questions. It was a quick 5-minute form where they just filled in the most necessary things that we then would probe more during the interview sessions.
2. Sending photos via WhatsApp for participants’ convenience
Learning from the previous diary study, some older participants had difficulties attaching the files to the online form. In this study, we decided to simplify the process by asking them to upload any screenshots they had taken and send them to us via WhatsApp. This simplified method apparently solved an overlooked issue of uploading files through Google Forms.
We just learned that to upload some files to Google Forms, the participants need to have some free space in their Google Account storage. They wouldn’t be able to upload anything if they had full storage. We were lucky! We discovered this thing in the middle of the interview session. One participant told us,
“It’s a good thing to submit the screenshots via WhatsApp. I have no space left so I won’t be able to finish the task you gave me.”
The Perfect Diary Study Setup
These two projects taught me there was no perfect diary study setup. It all depends on you and your team as the researchers. When creating a diary study setup, we need to consider 2 main stakeholders: the researchers who will collect and work on the data and the participants side.
For the researchers
- Map out all the processes from the preparation to the end of the study to plan an effective workflow. Remember the nitty gritty that is possibly time-consuming.
- Do you need to filter out the data? Do you need to translate the data? Try to design the process so it can ease you up, especially if time constraint is also an issue you need to tackle.
For the participants.
- With the participants in mind, pick the most familiar tool as the study platform. A familiar tool eliminated the possibility of a dropout because of a hard-to-navigate platform.
- Ensure understanding of the instructions and facilitate them with a channel to ask questions during the study.
With these lessons learned, you are set to embark on a diary study prep! It might be daunting but if you prep the logistics well, you can nail it.
Have you done any diary studies? Anything you’ve done differently?
I’d love to hear your thoughts!
Read my previous articles:
- Behind the Scenes of Pediatric Dental Study.
- Lessons Learned from Interviewing Vulnerable Participants
