Tips on how to run remote usability testing
Insights from working during the Covid-19 pandemic.
Usability testing is probably the least glamorous, yet extremely important aspect of user research. We were recently forced to work from home for an extended period of time and therefore switched over to conducting all our usability testing remotely. This is what we’ve learnt 🤓
The Do’s
Run at least one pilot test
Before sending out your invitation to hundreds of potential participants, it’s always a good idea to test the process end-to-end with one or two of your colleagues. This will not only tell you whether your instructions and tasks are worded intuitively, but also give you an idea of how easy (or hard) it is to join the session by installing the respective user testing software (Zoom, Lookback, etc.). Getting this feedback for a range of devices other than your own, can give you valuable insights into potential issues that may arise while setting up or during the session. Based on your pilot study feedback, you can optimise your recruitment requirements (e.g. only recruit users with a particular OS), tweak your installation instructions and fine-tune your tasks or user scenarios.
Be prepared to solve technical issues
One of the most important realities of remote user testing is that you are not in control of the participants’ setup. It’s a much less standardised approach than in-house testing. This is good and bad. It’s a realistic scenario but at the same time, it can bring up issues you have no control over. Some participants will forget they needed a headset, someone’s battery will die halfway through the session, some will struggle with bad wifi connections and yet others will receive calls which can kill the camera you needed to continue your video call. Ultimately, there is little you can do about this, apart from sending out a comprehensive briefing beforehand. What you can do, is to anticipate what problems may occur, be prepared to troubleshoot and have researched the solutions beforehand (e.g. if the connection is interrupted, have shortcuts available to quickly bring the user back to where they left off, troubleshoot audio issues, etc).
Invite design team members to join the session
Some remote testing tools allow observers to join a session without being noticed by the participant. This provides a great opportunity for the design team or clients to join in spontaneously and without interrupting. Equally, team members can choose to attend only a part of the session they are specifically interested in (e.g. when testing a feature they have been working on), making it more feasible for anyone to attend more frequently and to get a feel for the usability of their designs first hand. However, having your team or a client watch a session, doesn’t replace you and/or your team collating the findings and providing prioritised key take-outs afterwards. Time stamped notes (e.g. in Lookback) are a great way to link back to specific insights that the design team can refer to when brainstorming new solutions.
Mind your users’ privacy
If your participants are using their own devices (which is likely when you are testing remotely), make it your priority to respect their privacy. While incoming messages and other disruptions are a good reflection of how the product or service would be used in real life, receiving personal messages or notifications can get awkward for both the user and the design team. To avoid compromising the users’ privacy, make sure you inform them of the recording beforehand and prompt them to enable the “do not disturb mode” on their devices. Similarly, if you are planning on uploading images or other files, give your users a heads up and the opportunity to censor the content of their photo libraries before it’ll be viewed by you and your team during the testing session.
The Don’ts
Don’t create unnecessary work for your participants
Some formal pre-session requirements such as research consent and NDA forms are fairly uninteresting and often get neglected or ignored by participants. When testing remotely, it is important to make signing these as easy as possible. So don’t ask anyone to print a document, scan it and email it back to you. Rather set up an online form that participants can e-sign by setting checkmarks next to the respective paragraphs. Whichever way you choose to solve this, streamlining the process and asking your participants for as little extra work as possible increases your chances of running a successful test.
Don’t rely on participants being prepared
Even with the best briefing email, some participants will likely miss some of the details. Although remote testing can be less time intensive and more flexible than in-house testing, your participant’s preparation and readiness for the test play a much bigger role for the success or failure of your study when testing remotely. Don’t wait until something goes wrong, but rather send out reminders the day before or pick up the phone to do a quick pre-briefing on the day.
Don’t forget to double check your own tech setup
While you are busy reminding your participants of the tech requirements to join your session, don’t forget to get your own equipment ready. You don’t want to have to go looking for your headphones or your laptop charger during or just before the session.
Don’t overestimate the number of scenarios that can be tested
Just like you would for in-person testing, time-boxing and structuring your testing session is essential. However, you probably won’t be able to cover the same number of tasks or questions as during in-person testing. As is true for online meetings and workshops, your users attention span is shorter when usability testing is run remotely. Additionally, remember to plan in some extra time for troubleshooting the technology (if necessary). So generally, a good rule of thumb is to reduce your testing time by one third and adjust the content you want to cover accordingly. This will motivate users to stay on topic and avoids that the session runs over.
Don’t hide yourself and your observers from the participants
Factors that make your participants nervous, or self-conscious can be detrimental to getting useful feedback and insights. With remote testing, a higher level of uncertainty is inherent to the setup compared to when researcher and participant are in the same room. To reduce the impact of this unknown element, turn your camera on and take some time to introduce yourself and the study to the participant instead of jumping straight into the research. Even if you are doing unmoderated remote testing, it can be beneficial to share some quick facts about your team upfront, as well as to disclose what team members might be watching the session later. Users often prefer transparency to the unknown and knowing who they are dealing with will generally make them feel more comfortable.
Looking for a UX design agency to partner with?
We hope you found this article useful. Let us know how your remote testing is going!
We’re on a mission to empower organisations with strong, informed design. We also do usability testing (in-house and remotely). If you would like to partner with us or receive our research & testing credentials — Get in touch.
Also be sure to follow our journey and connect with us on:
