Run Diary Studies Like A Pro
Diary studies give us the ability to collect context-rich data over a longer period of time.
Recently, we ran multiple diary studies and here are some of the challenges we faced while conducting the study and our hacks to mitigate them.
Problem 1: Participants not providing sufficient details
For a diary study to work, it should give you rich data about the behaviour you are studying. Often we have participants who provide sketchy, monosyllabic responses. For example,
Question: Describe your experience finding something interesting to watch on Prime?
Expected Response: I was in the mood for something light & fun. I started browsing but could not find anything appropriate so I gave up and saw a rerun of ‘Friends’ instead
Actual Response: It was OK.
We do not learn anything interesting from such a response. So, it is important to get rich description responses of participants experiences.
Solution: Building rapport with participants
- Do a formal onboarding of participants. Build rapport with participants by briefing them about the goals of the study and the importance of getting detailed responses.
- Participants are more likely to give details on a phone call than when you ask them to write it, so do not hesitate to pick up the phone and do a follow-up call. During the call probe to get a detailed understanding of the participants response.
- Create a per participant cheatsheet containing notes from all the interactions with the participant. This helps you guide participants and nudge them for more details without being intrusive.
Problem 2: Participants unable to understand the question
Sometimes it’s the participant who doesn’t write enough. But sometimes, it is the question. Recognising that a question is difficult for a particular set of participants is critical in getting the exact data you need.
Solution: Rework open-ended questions
In order to recognise the issue, you need to spend some time at the end of each day to go through all the responses.
For example, if your original question to understand feelings about information consumption on social media is:
‘How do you feel about the social media platforms you commonly use for information?’
And it fails, where no one understands the question, it might help to rework to something like:
‘What do you think about the information you receive on social media platforms? (E.g. is it too repetitive, are you happy with it, or do you think it can be better?)’
Problem 3: Drowning in Data
The amount of data you get from a diary study can quickly snowball, especially if you are collecting data multiple times a day for a medium to large sample size study spanning a week or more.
Solution: Keep upto date with the data
There is no way one can keep up with all the data unless you start looking at the data and start analysis early.
“That was the big thing I learned: I’ve got to keep on top of those entries as they’re coming in to get the most out of the data.” — Sara Cambridge
Problem 4: Time Consuming
Diary studies are time-consuming. It may not seem like a lot of work; Send a form, wait for responses, follow up. But it is taxing and you have to constantly stay engaged.
I am the kind of person that leaves my phone all over the place. During the dairy study my phone was glued to me because I had to be available for our participants in case they ran into any issues, and it felt exhausting. Between following up with participants, onboarding, ensuring they reply on time, evaluating data, I found myself wearing thin.
As you get used to diary studies, you begin to fine-tune the way you handle situations and start understanding the value and beauty of the method. At the end of the day, it is a great tool for understanding human behavior.
