Remote User Research

Note: This post originally appeared in UXmag.com

We all want to create products that make people’s lives easier and more enjoyable, products that delight people around us and ourselves. Creating great experiences starts with understanding your users, their behavior, motivations and goals.

Remote user studies are ideal for getting insights from end-users who may be located anywhere in the world. They provide access to a larger pool of potential customers, cut out travel time, and can significantly lower the cost of usability testing.

In this article I would like to discuss methods and tools for effective remote UX research and how to make sure your research findings are put to good use.

When remote study work

Remote user research has two key advantages over traditional testing:

• Flexibility. The remote method allows us to connect with your customers anywhere in the world, give us greater user diversity and also can be quicker to set up.
• Environment. User interviews are conducted in the participant’s usual working environment with the hardware and software they use on a day-to-day basis or while they are on the go.

When remote study doesn’t work

There are a lots of reasons you might want to conduct an in-person study instead.

• Info security. When you want to keep control over confidential test materials
• Inability to use screen sharing. Some studies may require you to talk to users who don’t have reliable high-speed internet connections
• Face. The importance of seeing the user’s body or face, or track eye movements

Do a lab study when you need to use special equipment, keep the interface 100% secure, see the user’s physical movements , or when you can’t use screen sharing tools.

If you are a visual thinker like me you may like my Hand Drawn Design Tips — weekly 1 min read newsletter on product design.

Moderated vs automated research

The remote research can be divided into two categories: moderated and unmoderated research.

Moderated

In moderated research a moderator speaks directly to the participants. Moderated research allows you to gather in-depth qualitative feedback.

Since they provide lots of context and insight into exactly what users are doing and why, moderated methods are good for research when you’re looking for new ideas to come from observation.

Automated

Unlike moderated research, automated research does not involve any conversation between the researcher and the participants. Instead you use online tools and services to collect information automatically.

You can conduct automated research to get feedback from a large number of participants on a specific set of tasks. Automated research is usually quantitative and good at addressing specific questions like:

• what % of users can successfully log in?
• how long does it take for users to find a product?

If all you need is performance data, and not the reasons users behave the way they do, then automated testing is for you.

How to design a remote user study

In general there are no huge differences between planning a standard or a remote user study:

1. What. You need to decide what you want to learn. Before starting the study, everyone on the team should agree on the questions you plan to ask and the assumptions you plan to test.
2. Who. You’ll need to carefully recruit people based on your goals: existing customers, prospective customers, or representative customers. I really like a collaborative approach here when you. with your team, identify goals for the study, and list the characteristics of the people you want to interview and people you want to exclude from your study. Then figure out precise criteria you can use to identify those people and write questions to screen them.
3. How. Determine the best research methodology for a particular study that will allow you to reach your goals and write an interview guide or that keeps you on track during the interviews.

I would like to focus on “Who” and “How” steps and discuss what is different when you plan and manage a remote study.

Recruiting for web or mobile study

Live recruiting

It is really important to understand the usage context of an interface — the physical environment where people are using a product. And remote research allows you to instantly recruit people who are right in the middle of performing the task you’re interested in.

It is called Time-Aware Research.

Time-awareness means that users are interested in what they’re doing because they’re doing it for their own reasons, not because you’re asking them to.

For more details on Time-aware research you can read Nate Bolt’s amazing book “Remote Research”. It also has a lot of practical tips and templates that you can apply in your remote study.

Highly recommended!

A quite effective strategy for live recruiting is to use a popup form that will open the recruiting screener, which users can fill out to opt-in to your study. Or you could place a link to your survey in some noticeable spot on your site. It’s easiest to make by using Google Docs’ or you can use tools like Ethnio.

Ethnio is a customizable screener that can be placed as a pop-up on your site or accessed through a direct link to an ethnio URL. Ethnio provides a 2-line JavaScript code, which you should place on the page where you want to intercept your users.

It will take some trial and error to determine when exactly to show the popup and the right balance between session length and incentive.

Ethnio also works to find participants in real-time from mobile apps.

You need to place code from Android or iOS libraries into your app, then trigger an Ethnio screener based on actions you define in your app.

Another approach that works - just send a push notification to the selected audience with the proposition to take part in the study.

Other channels for recruiting

Of course you can use other channels to recruit someone for a study. Use agencies when you have higher budget; Usertesting screens people very thoroughly; Facebook’s or Craiglist’s ad work in some cases too.

But live recruiting is the best if you have an existing site or an app.

Incentive

The amount you should offer to the participants depends heavily on whom you are looking to recruit.

The average participation incentive for a typical remote study is somewhere between $35 and $50, which is lower than most in-person studies. If you are testing doctors or lawyers you will need to hike up the amount.

On the other hand, there are plenty of people who are willing to donate their time for no incentive at all.

This is especially true for services with loyal users who are quite often eager to participate in the study knowing that they may bring up problems they have encountered with your product.

There is a great article by Charles Liu, design researcher from KISSmetrics, sharing his experiments with recruiting people via emails with a zero budget.

One of the tips he gives is to change the conversation from “help me” to “could this help you?”.

In the article he is sharing several email templates that you can use for recruiting people for your own studies.

Tools for remote UX studies

Lets see which tools work for the purposes of remote research.

Moderated. Screen sharing & recording

Very often in the screener form I ask the participants what screen sharing service is convenient for them. Often Skype is the first choice. Sometimes they are not familiar with the screen sharing and in this case I suggest what better works for us.

For the remote studies at Stanfy we usually use the following tools:

• Skype — you can observe the face of the participant and the screen.
• Google Hangout —you can observe either screen or face.
• GoToMeeting ($49/month) —you can observe face and the screen and also it can give keyboard and mouse control to the participant.
• TeamViewer — sharing your screen with the participants and allowing them to play with the prototype.

GoToMeeting and TeamViewer enable you to view your participants’ screen and also allows your participants to see and even control your screen. This feature is very useful when you want to test a mobile app launched in emulator.

For the screen recording you can use such tools:

• Camtasia Studio (Mac, PC), $150.
• IShowU HD (Mac), $60
• QuickTime (Mac), Free

First two solutions may be a bit pricey but they will cover all you screen recording needs. They have nice video and audio recording options to record portions of the screen or windows.

Automated. Usability study

There are a number of automated tools that prompt users to perform a task and then record their behavior. If you want to learn how users interact with some part of your website or app, task elicitation is the way to go.

It’s a flexible way of getting a specific answer to common usability issues, such as the following:

• do users have difficulty performing the task?
• how many people abandon the task and where?
• what is the most common path users take through the interface to complete a task?
• where do users go to perform the task and where do they expect to go?

Loop11. A browser bar prompts users to perform a task. Automated analysis includes task completion rate, time on task, first click, click stream, most frequent success and fail points.

UserTesting. Users’ think-aloud comments are captured and synced with videos of their site behavior as they perform your tasks. Videos are downloadable and can be annotated with comments. You can also see users’ demographics and system info. Users are recruited by UserTesting.com from a preselected panel.

Moderated or automated mobile studies

Lookback. It helps you learn how people are actually using your mobile app. It records the screen of your app, along with the face and voice of the user, and all their taps and touches. With recordings like these, it’s super simple for anybody to send you feedback or bug reports, straight from within your app.

Lookback for Android records everything on screen, including any native app, mobile website or prototype.

Once a record is completed and uploaded it appears on Lookback’s website where you and your collaborators can view, discuss, rename and share them.

Unicorns.io. An app that streams whatever you’re doing on your phone — playing a game, texting, swiping through Tinder. Unicorns allows you to conduct usability tests, showcase pre-release apps or upcoming features and chat with the users live.

PlaytestCloud. It is an online service for crowd sourced testing for mobile games. The video recordings show you the player’s screen during their playthrough and highlight their touches on the screen.

You can discuss and share the videos inside your team.

Invisionapp. Allows you to transform your static screens into clickable, interactive prototypes complete with gestures, transitions, and animations. You can share a link for with your users to open on their desktop or mobile device and conduct usability tests with the users live. Invisionapp is great to quickly demonstrate concepts, collect feedback in one place and make rapid iterations.

Usertesting. I already mentioned this tool before in relation to un-moderated web studies but this tool works great for mobile studies as well. You can watch users complete the tasks you specify (including gestures), and listen to them speak their minds as they interact with your app. Their mobile recorder works with whatever is displayed on the mobile device, including wireframes and prototypes.

Observers

Remote research allows for a large number of observers who are also at their computers.

Having developers, project managers, marketing people, observe and engage with the research can make them more invested in the outcome of the research. Often that makes it more likely the findings will be applied.

Including everyone might sound like a waste, but it’s very important.

Making notes

While watching the videos, the goal is for the team to capture insights and ideas from what the user is saying and doing. The tools we use to do this are:

• post it notes in three colors;
• markers;

Instruct everyone to capture two kinds of things on paper while watching. Important insights (good and bad) learned from the user on the different post it notes.

Let’s say yellow for good and pink for bad. Any specific implementation ideas on green post-it notes. Ask them to write each down on their post-it notes using a thick, black marker.

Combine everyone’s insights

After several remote sessions gather near a white board and ask everyone to put their post its on the wall. The purpose of this step is to distill everyone’s notes into a common list of insights.

Once data is collected together with the team try to find trends, patterns, and themes that are relevant to the research question, and discuss them.

Do it all together — get the goals and questions from the team, then do the up-front work conducting a survey or getting participants to the interview. But the team should be there during the interviews, usability tests, and debriefing sessions, so they can observe users with their own eyes.

We collaborate for a day — during the research and after. We ask all observers to write down what they see and notice on the sticky notes, fill up the whole board with the sticky notes, and we collaborate to make sense of it. We help people to come up with their interpretations of what happened.

This is what we do now to make sure our remote research findings are put to good use and this how our report looks like.

Remote collaboration

If you are a visual thinker like me you may like my Hand Drawn Design Tips — weekly 1 min read newsletter on product design.

One really important aspect of a collaborative process during your research study is that if any member of your team is remote then everyone should be remote.

We did a lot of experiments with this, trying to involve a remote person via Google Hangout or GoToMeeting, but it never worked. Sooner or later all discussions start to happen between people who are in the same space, ignoring those who are outside.

There are many tools that you could use for the teams remote collaboration, we tried these:

• Realtimeboard
• Boardthing
• Murally

Summary

Here’s the quick 6-step summary of a test setup:

1. Choose a research method based on what you want to find out
2. Select the tool or service that best supports your research goals.
3. Create and post a recruiting screener and confirm valid participants.
4. Design a set of tasks or write an interview guide.
5. Conduct research session and pay out the incentives (don’t forget to include your team).
6. Summarize findings together with your team.

With an audience scattered across the globe, limited budgets and time constraints, the combination of different remote user testing provides the rich data that helps to make informed design decisions and to understand how people interact and behave with the interface you’ve made.

If you are a visual thinker like me you may like my Hand Drawn Design Tips — weekly 1 min read newsletter on product design.

I’m a UX Designer at @Stanfy — product design and development studio. We help others to design and develop software for a mobile and wearable world. Drop us a line if you need a technical or design help with your new and awesome technology product or startup.