Unmoderated Usability Tests for UX Research
What are unmoderated studies?
Unmoderated studies are done by participants following a set of instructions. Participants can complete the study any time, anywhere, and there is no researcher involved, so there’s no need to go through scheduling and the heavy logistics.
If you want to get a taste for how people design their studies, I recommend signing up to be a tester with websites like UserTesting.com. You get paid to test out websites.
Check it out here: https://www.usertesting.com/be-a-user-tester
PROS AND CONS
PROS
- Fast, easy, cheap
- Can run with many more participants compared to a moderated study
- You can get participants from all other the world or country, so your sample is more representative
CONS
- You have less control over the participant
- Can’t ask follow-up questions
- Can’t (shouldn’t) test confidential designs
- Unable to read body language as well
- You must write your tasks clearly and define when they are finished. Because if your participants don’t understand your questions, you can’t help them
Personally, I think the pros outweigh the cons. I love doing unmoderated studies given the right circumstances, but what are those circumstances?
When to use an unmoderated study?
When should you decide to do an unmoderated study vs. a moderated study?
It depends on a variety of factors, such as how much time and money you have, the complexity of the product, whether it’s confidential or not, or whether the research questions are lightweight enough.
- Tight on time and budget. The main positive of doing unmoderated studies is that it’s way cheaper and you don’t have to spend time scheduling people, or fighting for a lab space.
- You need a larger sample size. Since you don’t have to moderate, you can scale up your study more easily.
- When your product is out of the conceptual phase. Usually if you have a new concept or product and you don’t know much about it, I think it’s better to do moderated studies, because you can have more control, ask follow-up questions, and really understand the why. Consider unmoderated studies when your product is a bit more mature, like people could be using it in real life. You can get scalable insights this way
- “Lightweight” research. If you’re going to ask people about their entire livelihoods or want to understand a new uncontested market space, an unmoderated study probably shouldn’t be your approach here. But if you are looking for quick iterative usability feedback, this is when you want to do it.
Tools
I’m gonna list a couple of online testing tools that I’ve used personally for remote unmoderated studies. These are not free, but one of them has a free trial that you can try out.
Lookback — Free beta. Compatible with Mac, Android, and iOS and it’s a pretty popular tool these days. It simultaneously records facial reactions, screens, and gestures.
UserTesting, UserZoom, and Validately — they’re all pretty similar in my opinion, a few differences in pricing, but I’ve found them all reliable in terms of recruiting participants really quickly. I have launched unmoderated studies and they’re usually done in an hour or two, and that’s having 20–30 participants. It’s great.
You can do things like click tests, card sorts, tree tests, and surveys.
The great thing about these tools is that you can record people’s sessions and clip videos within the tools, so you can include these clips in your research presentations.
Steps to conducting an unmoderated study
The steps for conducting one is really similar to conducting a moderated study.
- Define your objectives and research questions.
- Open up your online tool and create a test, usually the online tools guide you pretty well on what to do. For our purposes I made a trial version on Lookback to give you an idea of how it works: https://lookback.io/n-a-37/projects
- For Lookback, you need to recruit your own users. Otherwise with the other tools like UserTesting they have their own massive database of participants, you need to create screener questions on the online tool itself, and you can filter for certain participants there
- Create your tasks, and be very specific about what you want the user to do
- Pilot the study with 1–2 people to test out any kinks
- Run it forrealz
- Download the videos and watch them, analyse and look for patterns
CONCLUSION
In summary, unmoderated studies are a quick way to gather feedback from your users. They can be done any time, anywhere, and I’ve used them when stakeholders needed answers to something in a day or two.
Want to learn more?
If you’d like to…
· learn all the details of Usability Testing
· get easy-to-use templates
· learn how to properly quantify the usability of a system/service/product/app/etc
· learn how to communicate the result to your management
… then consider to take the online course Conducting Usability Testing.
If, on the other hand, you want to brush up on the basics of UX and Usability, then consider to take the online course on User Experience. Good luck on your learning journey!
If you have questions, don’t hesitate to reach out kevin@zerotoux.com
If you are an aspiring UX researcher or someone looking to conduct user research in your everyday work, be sure to subscribe to the Zero to UX youtube channel for all things UX research!
Questions? Mock interviews? Resume/Portfolio reviews? Check out zerotoux.com
Mad love, peace!
