Async text-based research: an accessible methodology
Lessons learned from a case study
Experience research is about centering the humans in our technology contexts. For example, let’s say you’re asking people to reflect on their experiences using a tool that you’re testing. It’s helpful to speak to someone in real time to gauge their in-the-moment reactions. Yet, when they’re using this tool in practice, they probably won’t be using it in front of a live audience. Your users will probably interact with the tool on their own. They might mess around with it and maybe reflect on those experiences. In cases like this, we might lose some things by having asynchronous communications, but we don’t lose anything about the interactions itself. If synchronicity isn’t matching with the real world use case we are testing, it might not be a necessity when considering methods.
One way of centering the users and creating an inclusive research environment is to use accessible text-based interviews. These are interviews which take place in text dominant formats, such as via messaging or email. These types of interviews can and should be tailored to meet the needs of participants. When we’re planning interview studies, we don’t always think about the ways synchronous (or real-time/live) interviews might be limiting our participant pool, and in turn, the quality and depth of responses. Synchronous interviews have a lot of benefits, but not everything needs to be a video, phone, or in-person interview.
In situations like this it’s worth asking: would doing asynchronous (or text-based) interviews be a reasonable and effective accommodation? There are a variety of accommodation options, such as hiring a visual interpreter, using chat features, or incorporating live closed captioning. Text-based interviews are another valuable option you should consider when planning your next project. Based on my experience running a text-based interview for a workflow study, I’ve put together some considerations for incorporating such text-based interviews into your research toolkit.
Key terms
Before we dive into the case study, I’d like to introduce a few key terms. These are meant to help you understand what these terms mean as you continue reading.
Synchronous interviews: live interviews that typically take place in person, on the phone, or via video call.
Text-based interviews: asynchronous (not live) interviews that occur over a period of time, potentially hours or days. These can be call-and-response, where the interviewer asks a question, the participant answers, then the interviewer follows up. They could also be a list of questions, where the interviewer sends them over, and the participant responds to each in their own time.
Both types of interviews involve some dialogue between the participant and the researcher. Where they differ is the length of time between each response. Synchronous interviews occur live, and the participant and researcher will respond to each other immediately. In a text-based interview, it may take longer between each response.
The case: incorporating a text-based interview as an accommodation in a larger workflow study
Last year, I was planning a study at Adobe’s Content Authenticity Initiative to understand how various creators interacted with a new feature in Photoshop. To tackle this, I decided to run a workflow study, consisting of in-depth interviews with 11 creators. During recruitment, I asked all participants if they wanted or needed accommodation. One participant told me she was hard of hearing and English was her second language, and she asked if we could do her interview via written text rather than spoken word.
This is an important example of why it’s essential to“assume the person is an expert about their own accommodations”. If I had not assumed this, I might have directed this interview to include another accommodation, one that might not have been best for her or what she preferred.
My process for conducting this text-based interview included three stages: preparing the interview materials, conducting the interview, and analyzing the interview.
Preparing the interview materials
Simply translating the discussion guide into a document for someone to fill out isn’t accessible. This would generate more work for the participant, and it’s not a fair translation. Discussion guides are long, and are just that: a guide. Often, there are many questions which never get asked during an interview. As researchers, it’s part of our job to adapt this discussion guide to the individual interview. To translate this guide into a text-based interview, I had to think carefully about what was important. I asked myself: what did I really want this participant to respond to?
My final text-based interview consisted of set questions for a back-and-forth conversation via email and a word document containing the main task. This document had three sections: 1. a few core ethnographic questions to get the experience started; 2. prompts to engage with the existing Photoshop app, with specific questions to encourage reflection; and 3. prompts to engage with a prototype of a new feature, alongside reflection questions. Every prompt included screenshots of the relevant action, enabling me to communicate both verbally and visually the actions I wanted the participant to do. This document took on its own form, separate from the discussion guide, but still retained the essence of the core research questions the team wanted answered.
Conducting the text-based interview
Setting expectations is also important for text-based interviews. I outlined what the tasks entailed, approximately how long each should take, and provided space for feedback and questions. By signaling throughout the interview, participants will have an overview of the tasks and can allocate time accordingly. It is important to note that not everyone completes tasks at the same pace — let your participant guide this estimate.
Running the text-based interview was straightforward. We exchanged 3 emails where I asked her two questions and followed up on her previous answers. Once we had this exchange, I sent her the word document, and let her take the time she needed to complete the task. Throughout this process, I made sure to check in to see how she found the experience.
Analyzing the interview data
Finally, we must ensure that text-based interviews are valued just as much as synchronous interviews. I included the data from this text-based interview in my analysis the same as my other interviews, treating them all with equal weight. In the end, insights from this asynchronous interview were completely interwoven with the synchronous interviews. In my final reporting, there was no delineation for the two types of interviews. My goal was to create an inclusive reporting of my findings. This meant that I wanted to ensure all the findings were considered with equal importance.
Overall, we both found this to be a rewarding experience. The participant reported feeling engaged in the process and enjoyed providing her feedback. I gained invaluable insights into her experiences with the tool, and found the analysis process to seamlessly integrate with my other data points.
Concluding Thoughts & Core Takeaways
As this case study demonstrates, text-based interviews can be a valuable methodological tool. It’s important to consider accessibility and compensation when running a text-based interview, so that participants can have a positive experience.
If you’re thinking about incorporating text-based interviews in your research practice, these are some considerations:
- Consider if text-based interviews are an appropriate method from the start. If it is, consider noting in your recruitment that this is an option. This may help encourage potential participants who are hard of hearing or for whom English is a second language.
- Let the participant lead what their accommodation looks like. Only choose a text-based interview if this is something your participant thinks would be an engaging and helpful format.
- You cannot just use a discussion guide over email. Think critically about reformatting what the interview looks like. Simply translating the synchronous interview discussion guide for the text-based interview will unnecessarily burden your asynchronous participants.
- It’s important to think critically about compensation when giving people work. If you have users participating in both synchronous and text-based interviews, ensure you give them comparable workloads.
- Incorporate data from text-based interviews directly with your synchronous interviews, so that you aren’t putting more weight on one format over another. Both are valuable!
Text-based interviews are a great way to create an inclusive environment in research, and can generate valuable research insights. They can provide just as rich insights as synchronous interviews if done thoughtfully.
About the author
Amanda Curtis (she/her) is a freelance user experience researcher, focusing predominantly on immersive and emerging technologies. She is also affiliated with the Oxford Internet Institute, where she researches creativity and video game experiences.
