Remote User Research: What is it, how to do it, and why it might be better than in-person options

In the face of COVID-19, we’ve been seeing a lot of articles pop up on doing remote user research. In this article we’ll explain what it is, and entice you to continue using remote user research methods once we’re out of lockdown.

User research with real users is the most fundamental and irreplaceable part of UX. It provides direct information about how people interact with your product and/or service, and the problems and issues they face around what you’re building a solution for.

In-person user research is still the most widely used method of gathering feedback and understanding problem areas. This type of research usually involves letting users perform tasks on a computer while asking them questions, observing their behaviours and body language, or having them think out loud. When you’re doing research in person you have access to more dimensions of information on how your product is used and perceived.

In remote user research time and space are no longer as important, as participants and researchers do not interact in-person or face-to-face. The research is usually conducted via a computer or a phone, enabling the researcher to see or hear what the remote research participant sees or hears.

Planning your remote research

Before you start looking into which methods you’d use for your remote research, you first need to establish what you’re trying to find out and who you need to source this information from.

• What: establish what it is that you want to learn in the study. Before you start, everyone on your team needs to agree on the research questions, aims and the assumptions you plan to test.
• Who: create a recruitment criteria based on research your goals, to help you screen for participants — do you want to study existing customers, prospective customers, representative customers, a subset of customers? Are there any specific characteristics of the people you want to interview? Are there any people you need to exclude from your study?

Forms of remote user research

These next two sections will inform the “how” of your remote user research.

There are two overarching approaches to remote research, moderated and unmoderated.

Moderated research still requires the researcher to interact with the participant during the session, and is similar to in-person research.

This can be in the form of an interview, and/or thinking-out-loud exercises where the participant completes certain tasks while explaining what they’re doing, while the researcher can probe with questions and clarification.

An advantage of using moderated remote testing is that the participant gets to remain in their natural environment, which makes them feel more relaxed and honest about the experience.

Unmoderated research occurs without a facilitator/researcher. For this type of research the tests need to be pre-programmed and the researcher needs to set the user-tasks in advance.

Most often the participants are asked to complete a series of tasks, and their interactions are recorded. There can be a think-out-loud element to these exercises too, but there isn’t a facilitator to probe further or ask for clarification.

This method is useful when there are clearly described tasks which the users need to perform, with minimal room for ambiguity.

An advantage to unmoderated tasks is that users can complete them at their leisure, and won’t feel pressure while finding their way through the site/app.

Other unmoderated methods include surveys/questionnaires and A/B testing.

Remote research methods for different parts of product development

You’ll need to select different remote user research methods depending on which phase of product development you’re in, and what you’re trying to find out.

Discovery Phase

• In this phase of your project, broad ideas are usually gathered with in-person user research using workshop formats.
• Running workshops remotely is challenging to say the least- it’s harder to do collaborative, creative activities, and build trust and rapport within the group.
• In a remote environment, you might run a smaller online session and use collaborative whiteboarding tools, like Miro, to fill in the gap of post-it notes.
• Alternatively, you could do interviews to try and understand your users’ problems more intimately.
• Use thinking-out loud exercises to understand problems participants have with your existing product or competing products.
• An unmoderated approach would be creating surveys/questionnaires to gather data from a larger group of participants.

Prototype & Iteration Phase

• In this phase, you want to detect flaws in your design and approach.
• Remote user research in this phase shouldn’t need that much adaptation, and you can test your ideas from as early as you need to focus on and verify early ideas and basic features.
• Similar to the techniques mentioned above, you can create wireframes, mockups and clickable designs of your early-stage solutions, and use a moderated interview with thinking-out loud tasks.
• At this stage, you’d prioritize moderated over unmoderated, so that you can get more of an understanding of why your users feel a certain way about your idea, probe for alternative ideas, and really get a feel for their perception of the product. This can tell you whether you’re on the right path, and what to iterate on.
• Once you’re somewhat comfortable with the idea, you can go to a broader audience with unmoderated research, and pick up on usability issues. Verifying usability issues will help you to determine priorities for your next development iteration

Design Evaluation Phase

Usually, when evaluating your final product with in-person user research, you get high quality and detailed feedback about your product. There’s actually no reason why you can’t do something similar remotely

• Here you’d use moderated interviews and walkthroughs to gather detailed, in-depth information on the overall experience and confirm your product meets the needs of your users, and that there are little-no usability issues.
• Use unmoderated user-testing through heatmaps, A/B testing, and smaller, focused task completions.
• Remote testing is ideal for collecting feedback from a big sample of users enabling you to back up your qualitative methods with statistics data (answering questions such as, what percent of users can successfully log in? How long does it take for users to find a product?).

Tools and Resources to Run Your Own

In the previous sections, we introduced you to different types of remote research, and how you should take into consideration your development phase to decide what methods to choose.

There’s clearly a lot to consider when you’re planning and designing a remote test, and we’d need to write a hefty textbook to cover individual methods, tools and resources. So instead, we trawled through the internet and sourced some of our favourite guides to help you form your approach:

• A great blog on how to run a remote user research study, with lots of great diagrams.
• A 5-step process on running a remote usability test
• CareerFoundry have put together a list of tools for remote user research
• Remote Research has put together a matrix of tools, which is regularly updated, you can see which tools are best for which method, the price and value it provides. At the bottom of the page, they’ve aggregated their favourite picks.
• And this one from UX collective that looks at wrap around tools for all phases of your research

Benefits of Remote User Research

There are lots of benefits of doing remote user research (apart from being able to wear your pyjama bottoms while interviewing):

• They’re often a lot more affordable to run, and more convenient for the participant
• They’re much faster and require fewer resources
• They can be conducted in a real-world context, displaying more authentic use of your product
• It is much easier to create a diverse group of participants from all over the world, unlocking insights into cultural and contextual differences
• On that note, it means that people with disabilities are more likely to be able to participate, opening up access to more innovative product ideas
• It opens up an avenue of ongoing user testing, as the barriers to entry and level of commitment from participants is much lower

Tips for Remote Usability Testing

The observer effect is real! Technology seems to want to misbehave when the stakes are higher. Even if you’ve used a tool plenty of times before, test it, test it, test it… and then test it again with someone else.

Test everything from signing consent forms, the URLs to the call or tasks, through to the recording software. Participants are precious and you’ll be kicking yourself for not ironing out the technical aspects before running a session.

Pilot your study

Convince your Mostest Closest to be your guinea pig. Run your study like you would be with an actual participant. This part is critical, because it enables you to make tweaks and changes to your study to ensure it runs smoothly, your participants understand what you’re asking of them, and the exercises answer your research questions.

This is particularly important for unmoderated tasks, where the participants are relying solely on your instructions. Anything that can be misunderstood will be misunderstood.

Compensate Your Participants

It’s important to show your appreciation for their time, especially if they are existing or future customers. It also means they may be more likely to show up to participate and provide higher quality insights

Over-recruit Participants

I once ran a study where I had 90% no-shows. It was horrific! No-show rates for remote research can be higher than for in-person studies. It’s better to add a few more participant than you think you need. This also enables you to have access to more participants if some interviews/tests don’t provide much valuable insights.

Going Forward: Combining In-Person + Remote User Testing

Like with a lot of things this lockdown period has shown us that some of the ways we operate have lots of wiggle room for change. Perhaps we’ll be more open to the opportunities that remote research and testing can offer us.

There’s nothing quite like exploring and discovering ideas together in workshops, but a mixture of both approaches can complement one another to create a more solid body of understanding — opens up the opportunity to connect our products, services and ideas with a global and diverse community.

Thank you for reading! If you’re interested in finding out more about how we practice UX and software development at Propellerhead, you can reach out here.

You can take a look at our other stories here.