How to Run a Remote UX Lab
Due to COVID-19 we had to quickly change our qualitative research to a remote approach. Here is what we learned after 3 months.
User validation is vital when launching new and improved features or products. On location UX labs were one of our go-to-methods to ensure intuitive and successful customer facing products.
Due to COVID-19 we had to pivot to a remote approach in a fairly short amount of time. It required adapting our methods, tools and communication. It also meant tackling a completely new set of challenges and problems.
In this article we would like to share our experiences with remote UX labs, our approach, the problems we faced and what we’ve learned thus far.
Our experiences are based on three test cases we tested in the last months — all in form of UX labs. By UX labs we mean a mixture of user interviews and usability testing of a prototype. The first and second UX lab had 3 iterations to test and improve a user flow for a central feature of a mobility app. The goal of the third UX lab was to validate a customer profile for a new orientation of a mobility website.
While the objectives were different, the challenges for the teams were the same.
Here is what we’ve noticed and learned — 7 tipps that make planning, conducting and evaluating remote UX labs easier:
1. Don’t underestimate the GDPR (General Data Protection Regulation)
The GDPR has clear regulations in regards to data protection of personal data of test persons. Everyone and everything that comes in contact with personal data of the testers, is affected by GDPR guidelines. That means not only the people involved in the UX lab, but also the tools, clients and service providers.
We made sure to involve our data protection officer early in the planning of our UX labs. They set up order processing contracts to make sure our set up is GDPR-compliant. Furthermore, we developed reusable legal templates for different data processing scenarios, to save time and money at future UX labs.
But being GDPR complaint on paper is only half of it: One thing we learned is better be safe than sorry when it comes to GDPR. Make sure all involved stakeholders and team members know exactly how to handle personal data. Errors can have serious consequences.
2. Always recruit extra testers
No matter how good the incentive or preparation of the testers, there are always unforeseeable circumstances that can lead to testers not showing up, or not working out. Therefore, we usually set up extra interviews with alternate candidates in the late afternoon of the test day. The number of back up interviews depends on the total number of tests. A tried and tested ratio for us is 1 back up for 5 regular interviews.
3. Give clear instructions for the tech set up of the testers
Having to update old browser versions to make the required web apps work or — even worse — dealing with a slow internet connection and the resulting bad audio and video quality. These kinds of problems always result in losing a lot of valuable interview time. Often in vain. In the end we often have to cancel interviews despite everyones efforts — leaving all parties frustrated and annoyed.
We’ve learned to save ourselves some time and nerves by being more specific about the needed technical setup of the test person. Whether you recruit the testers yourself or contract an agency, make sure to state the minimum requirements for the tech setup.
Unfortunately, we still haven’t found a way to avoid smaller technical hick ups like camera or microphone issues. Our workaround is to have our tech experts in the call for the first couple of minutes. They help the test person to set up everything correctly, so the interview can start and run smoothly.
4. Pay attention to the choice of tools
The tool you use to communicate remotely with the test person needs to have a lot of qualities.
For the test person:
- they have to be easily accessible
- the set up has to be quick and easy
- they have to be intuitive to use.
One thing we learned is that the GDPR complaint tools, are sadly not the most user friendly ones for the testers. A workaround that helped us was to send a step by step guide how to set up the video call tool in different browsers. The tech set up with the expert at the beginning of the remote call also helps in these situations.
5. Choose the right tool to share your prototype, especially if you want to test a smartphone app
There are two ways of testing smartphone apps remotely.
You can test with a smartphone mockup on a website. This version is super easy for the testers. They get a link and open a website with the prototype on their desktop device — no need to install anything. We like to use protopie cloud for that.
The other option is to use a special software e.g. lookback, so the test person can test the prototype on their own smartphone. This is a bit more complicated. It requires installing the lookback app on the test persons device. You also need to make sure that the prototype works on different screen sizes and ratios.
For our UX Labs we chose the first option to test with a smartphone mockup on a website. While the setup was straightforward, we noticed that the scrolling behavior inside a mockup does not feel natural to the user. The effect is that the users don’t really explore the screens below the first viewport.
We learned to keep that in mind when choosing prototyping tools for future UX labs.
6. Decide before the UX labs where and how you want to make notes and document the results
In the past we were used to conduct UX labs on site. The team members or clients observing the UX labs would usually be in the next room, taking notes on different colored post its. At the end of the day one wall would be filled with insights. We would discuss them all together, extract the key insights and decide the next steps together. Obviously that isn’t an option during the pandemic. Sadly, we didn’t initially realized that this might be an issue. Hence, in one of the first remote UX labs we did not clarify where and how to take notes during the interviews. We also didn’t talk about how we wanted to document the results to share with our clients and team members. The result was that everybody used a different tool and the notes itself were vastly different and hard to compare. That caused a lot of extra work and effort.
We learned from that and created a template for a shared notes document. Everyone taking part in the UX lab can simultaneously write down their observations. The key questions are stated on top of the document, so everybody knows what insights to look out for. We also made good experiences with the rainbow spreadsheet format. It makes the evaluation and extraction of key insights a lot easier and faster.
7. Don’t test on a Friday
We know, Design Sprints usually do user tests on a Friday. But hear us out. When not only the test person is interviewed remotely but also you and your team work remotely, the break down of all the notes, the analysis and the documentation is more strenuous and takes a bit more time. Trying to finish it all on the interview day didn’t really lead to the best result.
Therefore we make sure that we block the morning of the following day. This way everybody is rested while the insights are still fresh. We take this time to look over the analysis from the day before, extract the key learnings, formulate the next steps and document the insights.
We noticed however, that having a weekend between the interviews and the final evaluation made it a lot harder to remember the test persons and things they said or did.
These are the main learnings that helped us optimize and streamline our remote user interviews thus far. We are sure there are a lot more tipps and tricks and we will continue to learn and tweak the process with every new remote UX lab and test objective.
Please share your remote UX lab experiences in the comments. Maybe you have some learning to share with us?
For more of our remote working hacks, check out our article regarding remote workshops.
Written by Helena Rott, UX Designer at DAYONE
About the author
Helena has a background in product- and interaction design and joined the DAYONE team as a UX Designer after and interlude in VR research. Getting to the bottom of user problems with the help of qualitative and quantitative research is the best part of UX Design in Helenas opinion. In her daily work she particularly enjoys working with other disciplines, optimizing team and work processes.
What else does she like? Interior- and fashion design and obsessing over random British and Scandinavian TV shows are some of her favorites …
