Designing an Effective Research Protocol
Identifying biases and developing a sequence of events, activities, and questions to encourage trust and foster an open dialog with participants
While research into prior work in this area was underway, we started in on our own protocol to develop our understanding of the space — what was working, what wasn’t, and what that would mean for an autonomous vehicle experience in the relatively near future.
Goals
We sought answers to some foundational questions to not only develop our own understanding of the problem space, but also develop our own sense of empathy towards our target users. Our intention was to establish a baseline around which we could build our design work and draw inspiration. Looking at what we were finding in prior work review and what we knew we’d need clarification on, we found that we had a pretty good list of high level questions:
- Current transportation usage: What transportation services are currently used today by people with visual impairments? Why are these services used? Where might the friction points and successes be in their experiences?
- Friction points: Where might the friction points be in the rideshare experience as it exists today for people with visual impairments?
- Passenger information needs: What external or additional information does our target user group require in order to successfully complete their trip?
- Driver interaction: What does the interaction look like between a driver and a passenger in this context?
- Mobile device usage: How is our target user group currently using their mobile device in-vehicle during a ride sharing experience?
We’ll dig into each of these areas, their importance and how we want to address them through the protocol.
Conceptualizing data gathering
We researched and brainstormed methods to use to gather the data that would be most helpful. Between interview formats, interactive activities, and observational methods, we discussed at length what direction would be feasible and most effective. In the end, we opted to do both phone interviews and in-person sessions that would feature an interview portion and an observation session in which we could observe the transportation experience of the participant.
Phone interviews would present a relatively low-barrier alternative to having participants come to us while presenting us with primary data points that we could use to start to answer the questions above. However, the in-person observation sessions would be critical for us to determine where in the in-vehicle experience any friction points might lay, any critical interaction points with the driver might be, or any additional needs we couldn’t foresee. The data gathered from an open dialog can be extremely valuable, but the data we could gather from witnessing first hand what our participants’ experienced would be crucial to determining where we might go with the design.
Of course, a major consideration with this line of research is bias. Throughout this project we’re concerned not only with the biases that surround our target user group, but our own biases in designing solutions for a population of which we’re not members. At the heart of each of our decisions is a desire to engage this population and include representatives throughout major decision points to help us ensure that our work addresses real needs and does so in a way that is realistic for this population to adopt.
Constructing a script
With our methods set, we sought to formalize a script for each session format that would address our research goals to some extent. Our process for this was based very heavily in those research goals, to the extent of essentially assigning each goal a set of questions and a set amount of time in a given interview or observation session. To illustrate a few example questions, we’ve broken down our process by goal:
Current transportation usage
What we need to know: Qualitative experience data around the transportation methods our target user group use today — what works for them, why does it work, what could be better — and quantitative transportation data to better understand usage at scale.
Methods we used: For the qualitative data, we used interviews, both over the phone and in-person. For quantitative transportation data, we looked to government agency reports (primarily U.S. Department of Transportation, Pennsylvania Department of Transportation and associated initiatives).
Questions we asked: We opened with a question around what modes of transportation were used in the last week, and then from there asked if this was representative. For each mode listed, we asked questions about the recent experience, what stood out as positive, what stood out as negative, and if there was something that a participant would want to change.
Friction points
What we need to know: Qualitative experience data around any potential difficulties, hardships, or problems within the transportation experience as well as methods for overcoming these friction points.
Methods we used: We used interviews, both over the phone and in-person, and in-person observation sessions.
Questions we asked: We asked questions around recent negative experiences related to modes of transportation the participants take and for the in-person sessions in which we could observe first hand, we were able to gather notes around areas that presented obstacles as well as strategies for working through those.
Passenger information needs
What we need to know: Qualitative experience data around the the types of information passengers with visual impairments need to have a successful trip.
Methods we used: We used interviews, both over the phone and in-person, as well as in-person observation sessions.
Questions we asked: Within the interview sessions, we strived to ask questions around what the sequence of events entailed in a particularly positive trip — what happened first, then after that, and so on to be able to identify any points in the journey where information played a role. Likewise with negative experiences.
Driver interaction
What we need to know: Qualitative experience data around what kind of interaction or even intervention from the driver of various modes of transportation make for a successful trip.
Methods we used: We used interviews, both over the phone and in-person as well as in-person observation sessions.
Questions we asked: In addition to the questions we were asking around the detailed sequence of a trip, we also asked questions focusing on the relationship with the driver, such as if the passenger talks to the driver, what kinds of things they talk about, what makes for a good driver, etc.
Mobile device usage
What we need to know: Qualitative experience data around the usage of mobile devices in the vehicle.
Methods we used: We used interviews, both over the phone and in-person, and in-person observation sessions.
Questions we asked: For interview questions, we asked about device ownership and usage during the ride, wanting to know what our target group might be doing while they’re in the vehicle and what they might need from their devices. In our observation sessions, we witnessed first hand what participants did with mobile devices during the trip.
Assessing the script
We piloted out draft script to see where we might have gone wrong, points of potential breakdown, and areas that could be more effective. With this particular protocol, given our ambitious goals, we wanted to make sure that we left as little up to chance as possible. A few areas that we developed even further in our final protocol:
Sequence of events and timing
In our original protocol, we put together a rough outline of the sequence of events for each format, from scheduling a session to starting one, to review of the consent form, and conducting the session itself. The biggest consideration for us was timing the in-person sessions to that we were able to maximize our time with participants and still have as much overlap in researchers as possible so that one person could lead the activities and one could take notes. Our observation sessions dictated that one researcher travel with a participant while one stayed behind to make the observation session as natural as possible. This meant that we needed to choreograph the start of the following session with the end of the previous one, so thoughtful planning was needed.
Introductions and warm-up
With our warm-up questions and conversation, we wanted to ease into understanding the fuller context of our participants’ transportation habits and needs. While we originally thought to ask about work, we changed to asking about what they do outside of the home instead. This allowed us to get a fuller picture of all that our participants do and need transportation for, but also got our participants thinking through frequent destinations and activities.
Observation over think aloud
For our in-person session, we initially thought a think aloud might be a good way to understand more about our participants’ thought processes as they used transportation. However, on review, we realized that this might make our participants uncomfortable and distract from a more authentic experience, making it more challenging to assess how they normally engage with systems and services. We opted for a quiet observation session and to only prompt questions as we felt they needed to be asked.
Snowball sampling
Recruiting participants is allows a challenge and for this population and location, it proved to be even more challenging. With this built into a protocol, we were able to leverage our participants to help recruit others.
Adjusting and improving
As we started our sessions, we learned quickly where we minor updates or slight adjustments could make things smoother and more efficient. For the most part, our goals were fairly straightforward, so no major revising was needed to the protocol as the research continued, but we did realize one key area that needed attention. While not uncommon, our compensation to participants was a gift certificate to Amazon.com, a popular online retailer. However, this proved problematic for a number of our participants who either didn’t shop with them or rarely shopped online. Fortunately, this was simple enough to adjust and we were able to provide those participants with satisfactory alternatives.
Next up
With this round of research gathering completed, we next looked at how best to distill our findings using a grounded theory approach.
