Going global with international research
As a London-based research team with a global remit, we’re constantly looking for ways to include other markets in our projects. So far we’ve tried a number of different techniques for international research, and each one has taught us a lot about what works and what doesn’t.
In this post, I’ve asked some of our research team (thanks Sophie, Struan, Audrey and Francesca!) to talk about the international research methods they’ve used and how they found them. This is a pretty long post, but hopefully some of the things we’ve learned will be useful to other researchers doing research internationally.
Cultural immersion ( Sophie)
Since we already had an awareness of some significant differences with the Dubai rider fleet, we wanted to validate them before continuing to build things for (what sounded like) an unfamiliar user. So at the end of last year we went to Dubai to immerse ourselves in the culture and come to a better understanding of the challenges they face.
Our aim was to gain deeper insight into who the riders are, as well as the environment they’re working in and their experience with the current product. We used a variety of research methods across the week:
- Usability testing the rider app with a native moderator.
- Shadowing riders while they worked.
- Interviewing the local team about their experiences of working with riders.
Advantages
By spending five days in the city, we were able to learn and feel things that would have been impossible from London. The challenges faced by riders included everything from hot weather and complex road networks to the vertical architecture and language barriers. All of these things could only be fully understood through first-hand experience and observation.
Understanding these aspects of the market allowed us to connect the dots, until all the anecdotes we’d heard in the UK fell into place. Only then did it become clear that the Dubai riders had some needs that were significantly different to those in the UK.
Disadvantages
There’s a huge amount of planning involved in organising a full week of research in another country. Being removed from the area geographically makes everything a bit more complicated. For the usability testing, an agency helped us to recruit, moderate and simultaneously translate our sessions from Urdu to English. However, only after arriving we realised that there had been a miscommunication about the recruitment, which led to uncertainty on the validity of the findings.
Luckily, we were in Dubai for four more days, so there was enough time to further investigate the behaviour and attitudes of these users. By the end of the week, we felt confident in our learnings due to all the context we’d acquired.
What we learned for next time
You can’t be too meticulous in your planning. Looking back on this project, I wish I’d done more desk research prior to taking off. I’m sure we could have recruited more accurately and therefore had more faith and confidence in our findings. This is something I’ll be very careful about in the future because once you’re there it’s too late. My advice would be to leave plenty of time to explore your options to ensure that you get maximum value from your trip. Never underestimate how much planning should go into these hefty international research projects!
Simultaneous interpretation (Struan)
The rider team recently rolled out a new feature to riders in Paris. This feature was a significant departure from the riders’ previous experience, so we wanted to understand how they adjusted to using it over time. We ran an 8-week longitudinal study, conducting in-depth interviews with the same eight riders every week. For this project, we chose to work with simultaneous interpreters to mediate the conversation while also hoping to build a relationship between rider and researcher over the time period.
Advantages
There was one clear and invaluable advantage of simultaneous interpretation over having a native-speaking researcher conducting the interviews. Having been involved in the processes of design and development, we had a deep contextual understanding of the product, which would have been very difficult to pass on to a third party.
Hearing participants respond in real time allowed us to put this understanding to work — we were able to identify and press further into responses that spoke to the study’s deepest interests, as opposed to sticking tightly to a pre-written discussion guide. As a result, by the end of the eight weeks our discussions with each participant had become, to some extent, personal to their experience of the product change. In turn, this informed how we represented and framed our findings back to the product team.
Disadvantages
We’d hoped that moderating the sessions ourselves would help us build a rapport with the riders over the eight weeks, but introducing a third party complicated the process of building a relationship. The interpreter’s presence was a constant reminder of the cultural differences that existed between participant and researcher.
My interview technique was also affected — whereas in a native-language interview I might pause after a line of questioning to leave the participant space to volunteer extra information, this proved hard to maintain as the presence of an interpreter altered the sensation of this silence, from natural to awkward.
However, the fundamental challenge was in the interpretation of a handful of Deliveroo-specific terms around which the study pivoted. These were often embedded in complex questions, with the meaning altered through subtle emphasis that could be missed by the interpreter.
What we learned for next time
Sensing the meaning of some of my questions was partially lost in translation during early weeks, I began to share more of the study’s academic goals and interests with the interpreters. This provided the context they required to register and repeat specific terms as intended.
My tip to other researchers attempting to do international research using this method is to maintain consistency wherever possible and include the interpreter as an equal actor in the intimate relationship that you’re working to build with the participant over the course of a longitudinal study.
Remote moderated testing (Audrey)
We don’t have easy face-to-face access to our customer service agents as they’re located in other countries. So if we want to do the research ourselves, remote research is really our only option for getting quick understanding.
As we needed to test designs for a new tool and explore the CS agents’ experience at this very early stage, we chose to do remote moderated research (facilitated method where you and the participants are not in the same room). Unmoderated research (unfacilitated method where the participant narrates an experience based on a set of tasks and questions) wouldn’t have given us the control or depth of information we wanted.
Advantages
Running the session ourselves meant we had much more control over exactly what happened and how. It also saved a lot of time and money compared to finding, briefing and liaising with an agency. We found set up to be quick and easy with the right tool (we used Lookback) — as long as there are no technical issues (more on this later).
Because participants take part by using their own equipment and in their own environments, we were able to collect rich, contextual detail that we’d have never gotten in a lab testing. You’ll get more ecologically valid feedback from this type of research.
Disadvantages
With any remote research, you can never be 100% sure of what the tech setup will be on the other side, no matter how rigorous you are with briefing and screening. We have a checklist of things we go through (do you have a webcam and a microphone that you’re familiar with? are you using a broadband connection? etc) but it’s not foolproof. Technical issues make things challenging as you lose a lot of time troubleshooting — or worse, the session may not go ahead at all.
Participants can also take part in the sessions wherever they like, so you have a lot less control over the setup of the session than you would in a lab, which can make running the session difficult. I once had a participant’s 3-year-old daughter wander into the room demanding attention. Another took part from the comfort of her bed, which was not so much inconvenient as a little unexpected!
Even when the environment is controlled, communication is never as good as face to face. There is sometimes a lag, which breaks up conversations, and it’s harder to pick up nuance and establish rapport over a screen.
What we learned for next time
- Get a phone number from the participant
This is invaluable for troubleshooting, especially when things go wrong with technology. - Be scrappy
Your first sharing/recording solution may not work so you might need to get creative and be flexible. Last time the tool we wanted to use failed and we ended up having to ask the participant to install Slack and use the screen-share function. We then recorded the screen using Quicktime. - Pay attention to the context
We learned a lot from observing users in their own environment and this can be incredibly rich and useful. Make sure you’re taking all of this into account in your analysis.
Because communication is a little more difficult, it’s important to establish rapport early. Do as much as you can to build a friendly relationship with the participant before the session and it’s much easier to get them feeling comfortable.
Remote unmoderated testing (Francesca)
We recently carried out some remote unmoderated testing of the website because we needed to answer a very specific question about conversion in France. Using this method meant that it was easy to turn the research around in a short timeframe, plus the risks were low given that we were trying to understand a specific pain point in the user journey.
Advantages
The biggest advantage turned out to be that this was a relatively cheap way to get high-quality videos we could then share with the team and analyse together. Unfortunately, many of the expected advantages turned out not to be true, as you’ll see below.
Disadvantages
We used a third-party company to recruit participants and administer the research, which we thought would enable us to move fast. Actually, it ended up taking much longer than we thought it would originally, mainly because the participants weren’t of a good enough quality to be able to draw conclusions.
We learned what participants were doing that was causing the issue, but we weren’t able to understand why they were doing this behaviour as we couldn’t ask follow-up questions.
What we learned for next time
If I were to do more research like this in the future, I would definitely take more control of the process. Besides using a specialist recruitment agency to find the participants, I might even be tempted to do a moderated session with a translator. Even with the additional set up, I actually think it would still have been much quicker than the back-and-forth we ended up having with the third party, and we would have ended up with much more detailed and useful findings.
We’re keen to continue learning about different approaches to international research. If you’ve tried an interesting method that we haven’t mentioned then we’d love to hear from you!
