Unmoderated Usability Tests vs. Moderated Interviews 101

Through this post, I want to highlight what are some of the important considerations when building an impactful usability test.

When it comes to usability testing, moderated tests have long been the standard in the industry. A UX researcher (sometimes designer or product manager depending on the stage and the size of the company) sits down with a participant, observes their behavior, and asks them questions about their experience, whether it be on a real product or a prototype.

However, unmoderated testing tools such as Maze and Hubble have gained popularity in recent years as a way to get more participants and more data without the need for a researcher to be present or schedule real-time conversations with testers. But what are the pros and cons of unmoderated prototype tests? Let’s take a closer look.

Quick demo video of how to create an unmoderated usability test using Hubble

Pros:

1. Convenience: Unmoderated tests are convenient for both participants and researchers. Participants can take the test on their own time, without needing to coordinate schedules with a researcher. Researchers, on the other hand, can run multiple tests simultaneously without needing to be present for each one.
2. Cost-effective: Unmoderated tests are typically more cost-effective than moderated tests, since they don’t require a researcher to be present for each test. This can be particularly beneficial for smaller companies or teams with limited budgets or with limited capacity for real-time interviews.
3. Large sample size: Because unmoderated tests can be run with minimal effort, it’s possible to get a much larger sample size than with moderated tests. This can help product and UX research teams identify trends and patterns in user behavior that might not be apparent with a smaller sample size.

Cons:

1. Limited real-time control: Unmoderated tests give researchers limited control over the testing environment, which can make it difficult to ensure consistent results across participants. Researchers can’t observe participants in real-time, which means they can’t ask follow-up questions or clarify any issues that arise in the moment.
2. Less context on the research: Because unmoderated tests don’t provide as much context as moderated tests, it can be harder to understand why participants are behaving a certain way. Researchers may need to rely on assumptions or guesswork to interpret the results. Providing a detailed script and a very clear instruction set can be a good solution to mitigate this problem.
3. Self-selection bias: Participants in unmoderated tests may be more self-selected than those in moderated tests. This can result in a less diverse sample and bias the results in unexpected ways.

Visualization of heatmaps and clicks on prototype frames through Hubble’s Unmoderated Test

In conclusion, unmoderated prototype tests can be a useful tool for product managers, researchers and designers looking to get more data on their product’s usability. For teams that need to move quickly and gather continuous data, unmoderated tests can be a great additive tool to supplement your research sample without talking to users directly through zoom calls or in-person meetings.

However, they come with their own set of pros and cons, and it’s important to weigh these carefully before deciding whether unmoderated testing is right for your project. Ultimately, the best approach will depend on the specific goals of your research, the resources available, and the preferences of your team; using a well balance combination of moderated and unmoderated studies will yield the best insights and an impactful product development process.

If you would like try unmoderated usability tests at your team, please feel free to contact us at brian@hubble.team. We would be more than happy to help or answer any questions.