Usability Testing: Moderated vs Unmoderated

Nick Babich
Mar 13, 2020 · 8 min read

Usability testing is an integral part of user experience design. It helps product teams understand how their target audience interacts with their product and what problems they face during an interaction. Most usability testing methods fall into one of two categories: moderated and unmoderated usability testing.

In this article, we will explore both types of testing and share practical recommendations on how to conduct each of them.

Moderated usability testing

Moderated usability testing is a usability testing technique that requires the active participation of a moderator (a real person who will facilitate the testing). The moderator will have solid experience in the field of usability testing and user research and they will work directly with the test participant, guiding them through the testing process.

Moderated testing can be done either in-person (in a lab or work environment, where a moderator can observe user interactions with a product), or remotely (via a video-conference call, during which a moderator asks test participants to share their screens).

Moderated usability testing in the lab. Image credit projekt202.

When to conduct moderated usability testing

Moderated testing is better suited for collecting qualitative feedback (times when it helps to find the answer to “Why?” questions, such as “Why did participants interact with a product in a certain way?” or “Why is a certain problem happening?”). For that very reason, it’s recommended to conduct moderated testing in the earlier stages of design — when a team is working on a product concept and wants to explore various solutions. Moderated usability testing helps the product team understand the mental model of a user and use this information to inform their product design.

Pros of moderated testing

Cons of moderated testing

Best practices for moderated testing

Define goals and plan the scope of testing

Knowing what parts of the product you want to test isn’t enough. You need to define key goals you want to achieve by testing your product with users and selecting tasks according to this goal. For example, if your goal is to improve the usability of a specific flow, it’s better to avoid vague goals (such as, “Our goal is to streamline the signup process.”). Be more specific on what you want to achieve (say, “Our goal is to reduce the number of dropouts during the sign up from 60% to 30%”). These specifics will help you to define the standards for your product and measure whether your existing product meets those standards.

Clearly define criteria for test participants

Test participants should represent your target audience. And for that very reason, you need to know what kind of people you want to hire. Make sure those you hire are in the right demographic and demonstrate the right behavioral traits you’re seeking. You can use the information from your user personas to define the criteria for test participants.

Follow the classic structure of moderated usability testing

Setting up a usability test involves carefully creating a scenario. The classic structure for a moderated usability test (a proven way of testing a product) includes the following steps:

As soon as you create a structure with actual tasks, run a pilot test with one or two of the test participants to ensure that this structure works well for them.

Select the right fidelity for testing

As a tool, moderated usability testing can be run during the product design process. In this case, the testing is conducted using a prototype, not a real product. While it’s possible to use low-fidelity and high-fidelity prototypes for moderated testing, it’s always better to test with hi-fi prototypes because such testing will give you a more realistic idea of how users perceive your design. When test participants see a prototype that looks like a finished product, they will evaluate it accordingly. On the other hand, testing with low-fidelity prototypes is valuable in the early stages of product design when you want to collect opinions about your design from real users.

Ask test participants to think out loud during the testing

When it comes to completing actual tasks using a product, it’s recommended to ask test participants to talk through what they’re doing as they’re doing it. When test participants think out loud as they complete tasks, it helps moderators to understand the rationale behind every decision, which reveals their motivations/perceptions.

Record the interaction on video

Both a user’s reactions and their screens should be recorded during testing. The video can be used during the results analysis phase to draw a conclusion from the research. It’s recommended that you share the video with the entire product team to make them aware of any problems users face when they interact with the product.

Take notes during the testing

A lot of things might happen during moderated testing, and it’s relatively easy to forget something. Thus, it’s recommended that you write down any interesting observations and facts that you can later use during the test results analysis.

Analyze results and present them in a report

It’s not enough to run a moderated test. You need to invest time in analyzing the results of testing. The report you create after the testing should contain the most important insights and help the product team and stakeholders prioritize design decisions.

Here are some recommendations for creating a report:

Unmoderated usability testing

Unmoderated usability testing is completed by test participants in their own environment without a moderator present. This technique is usually used to test specific parts of a product (specific interaction scenarios), rather than providing an overall review of the user journey. Unmoderated testing is typically done using special tools for automated testing. The chart below visualizes the difference between testing tools in terms of the data they help to collect.

Types of data collected by unmoderated testing tools. Image credit Kathryn Whitenton via NNGroup.

When to conduct unmoderated usability testing

Generally, unmoderated usability testing is better suited for collecting quantitative feedback (providing answers to the “How many?” questions). For example, this testing method is valuable when you need to identify how common certain problems or behaviors are among test participants.

Pros of unmoderated testing

Cons of unmoderated testing

Best practices for unmoderated usability testing

Create simple, crystal clear tasks and mock up a real test situation

Since there is no moderator that can steer participants in the right direction, the outcome of testing sessions will largely depend on the quality of instructions you provide to your test participants. For that very reason, every instruction, task, and question needs to be carefully analyzed to eliminate any potential for misunderstanding.

Test your usability test before inviting real test participants to take part; this will ensure that instructions are clear. If you don’t have test participants, you can try pilot testing your test on a person that represents your target audience but wasn’t involved in the designing of your product.

Recruit more test participants

No-show rates for unmoderated testing can be higher than for moderated usability testing. That’s why it’s recommended that you invite more test participants to take part in order to achieve statistically significant results.

Plan for post-testing activities

In many cases, it’s possible to collect some qualitative or quantitative feedback from test participants by reaching them afterward with a quick survey. Many unmoderated usability testing tools allow predefined follow-up questions to be built into the script. The questions can be shown after a user completes a certain task or at the end of the session.

Should I go for moderated or unmoderated testing?

To choose the right method of testing, you need to focus on the type of data that you need to collect to meet your goals. Moderated and unmoderated testing are not competing, but rather they are complementary; in many cases, you will need both qualitative and quantitative data to create a good product. For that reason, you need to treat usability testing as an iterative process. You can start with moderated testing with a prototype to better understand users’ behavior and ensure that users can actually use your design. As you finalize your design, you can then run an unmoderated test to see how it performs in real-world conditions.

Visit XD Ideas for more unique insights and authentic points of view on the practice, business and impact of design.

Join a free Daily Creative Challenge to sharpen you UI/UX design skills

Learn about Adobe XD, the powerful platform where teams collaborate to create designs for websites, mobile apps, voice interfaces and more.

Originally published at https://xd.adobe.com.

Thinking Design

Stories and insights from the design community.