Quick start guide to “Un-moderated usability testing”
One does need to facilitate a usability test all the time!
Mr. Moore was a UX guy at a pizza delivery restaurant which was in the process of building a mobile app for the ease of their users and more sales for the business. Mr. Moore was well aware of the rising expectations of the users with experience on app and so he wanted to test the app idea and its usability with a set of users.
But there were couple of challenges with his present project — tight budget and short timeline. Someone just suggested un-moderated usability testing. But Moore was worried…. because he had never conducted an un-moderated mobile usability testing before..
Mr. Moore is definitely a character in my imagination, but the situation is not an imaginary one.. This article will talk about a step wise approach to un-moderated usability testing and why one should go for it.
What is un-moderated usability testing?
It is a technique of testing that evaluates the usability of an interface or product; parameters such as
- the ease of use,
- efficiency and
- satisfaction customers have with the interface.
It is similar to in-person usability testing however participants complete tasks in their own environment without a facilitator present. The tasks are pre-determined and are presented to the participant via an online testing platform.
Why choose un-moderated over moderated usability testing?
1.More realistic feedback : With traditional usability testing in labs, you are putting users into an unfamiliar environment, making them use your unfamiliar devices rather than their own, and physically watching over their shoulders, directly interfering with their experience.
But with un-moderated usability testing there are obvious chances that the user will slip into his “normal mode”. And the feedback you get will be much more relevant to how real people are using your app “in the wild.”
2. Accessing an audience that is geographically dispersed : When the audience is dispersed, it involves a lot of cost, effort and time to either get to them or call them for a testing. A remote usability testing makes a lot more sense in such situations.
3. To get a large sample of user behavior data
A small sample of in person usability tests may be all that is required from a design perspective but if your stakeholders are used to seeing large samples then using big numbers may be simpler than trying to convince them of the value of the small sample.
Further, where a new design is critical for an organisation or will have will have a substantial impact, the confidence gained from a large sample study can be valuable.
4. To compare two or more products or interfaces
Unmoderated usability testing is perfect for benchmarking studies — comparing either competitor products or different iterations of your own product. The ability to capture large sample sizes means that statistically significant differences between interfaces can be identified.
Steps for un-moderated usability testing :
The un-moderated usability testing is a three fold process-
- Preparation for the test
- Sending out prototypes
- Analyzing the user behavior
Preparation for the test —
#1. To begin with, define the specific objectives for the test. It’s important to fully understand why the research is being conducted. This will help with recruitment, designing the questions and tasks, and provide a reference point for analysis and subsequent discussions later.
#2. Write the tasks very clearly. Because you won’t be present to help explain the tasks to the participants, you need to make sure the scenarios are clear and specific. Avoid adding extraneous information in a task, which may confuse the test participants. Having said all of this, the one thing I really want to emphasize on is — Keep it short.
Tip : Before you shoot the task to the users run a pilot run internally. This will eliminate any semantics or typo errors. Not just that, it will also help you understand if your users would understand your task.
#3. Recruit your users. Many of the tools let you choose participants from their pool of candidates, while some will let you add participants from your own list. Both have their upsides and downsides. What has to be kept in mind is that users should be as close to your user persona as possible.
Tip : Recruit more users…. No-show rate can be high. You also don’t know the quality of an un-moderated session until you’ve watched it. It’s better to add a few more users than you think you need in order to accommodate such problems.
#4. Select the right online tool for the test. There are a dozen tools on the internet that support un-moderated usability testing. But obviously one wants to choose the best one! Here are the factors that you should check for while selecting your tool.
- Records user sessions and shows time taken on each design screen. (A must have)
- Has an option of writing clear set of tasks
- Can test anything — paper prototypes, wireframes, mobile app prototype, website and everything else.
- Instant results
- Leave feedback in terms as comments as well
- Shows user behavior data in the form of heat maps and conversion funnels.
- Direct sync with majority of tools you are using.
CanvasFlip is an obvious choice for any designers and UX experts across the globe. (Sign up and setup your un-moderated usability testing)
Sending out prototypes —
Lots of planning done! It’s now time for execution. Share the prototype with the users via email or url. Ya this is a pretty simple step. There are a couple of things you should keep in mind during this time.
- Be available!
Have I lost it? It’s un-moderated testing! Accepted. But even for un-moderated studies, be available by email, if not by phone, as much as possible to help with any potential user questions. - Schedule your email as per time zones
The quarter marketer inside me speaks! Your users might belong to different time zones and sending the prototype link at one-time-fits-all will not work. Schedule the emails such that the user receives the email when it is 9.00 am — 11.00 am for him/her. This will ensure lower no-show rates.
Analyzing the user behavior —
Once all the users have taken the test, your next task is to analysis the user behavior recorded. To begin with, look at the metrics such as overall task completion, conversion rate, average time taken, average interactions etc..
With an overall feel for the product look into the individual tasks and identify those that are causing issues. Next you need to find out why. With video based tools, watch video of specific tasks to observe behavior to identify the elements of the interface that are causing the issues.
Next step is to analysis the cumulative data of all the users. This is in the form of conversion funnels and heat maps..
Over to you :
Un-moderated tests provide statistically significant usability results in a short period of time. And the beauty of the process is that one does not have to wait for any final interface. It can be done as early as on paper prototypes. So test often and test early!
Happy testing!
