My Experience Planning and Conducting a Remote Usability Study
Using Figma, Zoom and Telegram
The Covid-19 pandemic has caught us all off guard. Work from home has become the new normal, and we are now relying on telecommunications more than ever. Just like all other forms of interaction that has been taken online, usability testing is not spared.
Remote usability testing is here to stay, so an important question for UXers is: How can we make our study as seamless, efficient and as effective as possible?
Over the summer, I had the opportunity to plan and conduct a Remote Usability Study regarding different Two-Factor Authentication (2FA) methods for Silence Laboratories, a cybersecurity startup, in collaboration with SUTD-MIT International Design Centre. Having identified that existing 2FA like One-Time Password (OTP) is a pain to users, and that the less painful push-based 2FAs like the simple “Tap to Approve” are not secure, we were going to investigate if some new push-based methods (that were more secure than OTP) were as user-friendly as the existing “Tap to Approve” method. I will outline the process I took, my rationale for making certain decisions, and my learning points below.
Background
My study is a little complicated due to the nature of 2FA and the volume of prototypes I have to test
To begin with, this usability study was a little complicated due to the nature of 2FA. Laptop to phone 2FA involves the person logging in to a site using the computer and receiving a prompt, carrying out some actions on the phone, and then carrying on with the task on the computer. I was also going to do it using mock-ups, and I had 9 different methods to test. For each method, there were two prototypes — one for the laptop component of the 2FA and the other for the phone component. I also had to do 3 trials for each method (as we wanted to track learnability), with each trial having a different variation of the method. That gives a total of 54 prototypes!
Challenge #1 of my Remote Study
How can I observe the participant carrying out the action on both his/her phone and laptop without disrupting the flow or inconveniencing the participant?
While an in-person study might let me observe the participant carrying out the 2FA on both the laptop and phone components, doing the study remotely via screen share on Zoom would only allow me to observe one screen — the laptop or the phone screen. I could combine both the laptop and phone components as one prototype, and that will let me observe the process from start to end via screen share, but that will pose two problems:
- The participant will carry out the entire process on the laptop, or the phone, completely skipping the transitioning phase from laptop to phone, which is the “highlight” of all laptop to phone 2FA experience
- The participant will be carrying out half the authentication on the wrong device, for example carrying out the phone component on his/her laptop instead of phone. This might give the participant a very different experience and affect our findings.
Tackling Challenge #1
As the transitioning phase was a key aspect of the 2FA experience, and as we also wanted the participant’s experience to be as close as possible to the actual experience, I decided that we will just have to split the prototype into two parts — the laptop component and the phone component. How then, can we observe both screens?
My team and I brainstormed many ideas -
- Get the participant to join the Zoom call via both devices and screen share on both
But that will be too disruptive to the flow and we would also have to inconvenience the participant to download Zoom on both devices. After testing it out, we also realised that the same Zoom account cannot be in two calls simultaneously.
- Observe one component instead of both, and rely on interviews and surveys to understand the overall experience.
The downside of this is that without observing the participant’s interactions with the prototypes, we might not be able to ask the best questions to sieve out valuable information. Also, getting quantitative data (for example, the time taken to authenticate) would be more challenging.
In short, we wanted to find a way that will
- Mimic the actual 2FA process as close as possible — two devices, have to transition from one device to another
- Allow us to observe the process as much as possible
- Allow us to get the data we need without having to involve participants
After brainstorming and testing out different methods (only to realise that they do not meet our criteria), we finally found something ideal.
I will split each prototype into two components — the laptop and the phone component. I will use two laptops to carry out the study (thankfully, I have a spare laptop), and utilise Figma’s Observation Mode and Zoom’s screen share. On laptop A, I will call the participant via Zoom, and also share the laptop component of the prototype with the participant via my Zoom screen share. Before starting, I will also ensure that participants have their phone component ready with the prototype loaded on their phones (as the prototype takes some time to load and will disrupt the flow). On laptop B, I will have my Figma file ready at the correct phone prototype, and utilise Observation Mode on Figma to observe the participant’s actions on his/her phone.
Challenge #2 of my Remote Study
How can I keep the study as efficient and as participant-friendly as possible, with the large number of links that I have to send out?
Now that we have found a way that satisfies all of our requirements, the only concern I had was with the length of the study. From my rehearsals, sending 54 prototypes at different junctures takes time. It can also get confusing as the order in which the participants try the prototypes is randomised. I can pre-send all the prototypes, but that would be inconvenient for the participants as they will have to scan through the links to find the correct link. It would also be inconvenient for me if they were to accidentally click on the wrong link and try the wrong prototype.
Tackling Challenge #2
To tackle this, I decided to use Telegram, a messaging app that I knew almost all my participants would have installed already. I created a group chat for each participant and pre-sent all the prototypes and surveys into the group chat in the correct order. For example, trial 1 prototypes would be the most recent messages in the group chat, while trial 3 prototypes were sent earlier. For each trial, the sequence of the prototypes were also unique to each participant, and the prototype that was going to be tested first was the most recent message.
I also used the “pin” function to pin the message containing the links to the surveys. I only added the participant in the group chat during the study after I had given an introduction, so that the participant would not have played around with the prototypes beforehand. I used the “Delete for everyone” function on Telegram to delete the message containing the link to the prototype once the participant has completed that prototype.
All these meant that the participant only had to remember to click on the most recent message to try out the prototype, and click on the pinned message directly below the group name to do the surveys. This made things straightforward and also significantly reduced the time for each study.
On my end, I also had group chats for myself with the correct order of the prototypes so I could readily click on the links so that the participant would not have to wait for me.
Downsides of my Method
Having said so much about the steps I took and my rationale behind my decisions, the method that I eventually adopted was not perfect as well. Here are some downsides:
- To use Figma’s observation mode, the observer will have to click on the participant’s name to observe his/her actions. However, as the participant will be opening many prototypes over the course of the study, the participant has to be reminded to constantly clear previous tabs else the observer might be observing the wrong prototype and end up not observing anything at all!
2. The observer has to be very alert as there are many things to look out for — deleting the links to completed prototypes on Telegram, opening the link to the next laptop component prototype, opening the link to the next phone component prototype on Figma, ensuring that you have clicked on the correct name to observe, taking down notes and results. It gets better after the first few participants though.
Learning Points
I’ll end off with 3 main takeaways from my remote usability study experience. Perhaps you can consider these before you embark on your remote usability tests :)
- Planning begins at the prototyping stage
It is important to consider which prototyping platform best supports your remote usability study. I knew that I needed a platform where I can share prototypes easily with non-team members, and also observe participant’s interactions with the prototypes remotely, so Figma stood out for me.
2. The importance of conducting “full-dressed rehearsals” before conducting the study on participants
We might sometimes assume that we are already very familiar with the online tools, and think in our heads that some solutions work, like “Let’s get the participant to join the call from both the phone and laptop!”, but we won’t know that it doesn’t work until we have tried it.
My “full-dressed rehearsals” also helped me to identify problems with my study flow and allowed me to think of solutions to address it before the actual study. Realising the problem during the study itself might have caused me to “sacrifice” the results of the first few participants, and might also have resulted in a negative experience for the participants.
3. Keep the participants in mind when designing the study
I strive to give the participants the best possible experience during my study, as they are doing me a favour by participating in the study. This makes me think of ways to make my study clear, efficient and smooth. By keeping their needs in mind, I realise that it sometimes pushes me to think of more creative ways of dealing with problems, and this in turn benefits my study as well.
My experience planning, conducting and subsequently analysing the results of this remote usability study has taught me a lot about user experience research, and has also allowed me to apply my skills (statistics, data analytics, design-thinking and systems thinking) that I have picked up as an Engineering Systems and Design undergraduate at SUTD. Although this blogpost only documents a fragment of my journey, it was a good opportunity for me to reflect on what went well and what didn’t. I am very thankful that Silence Laboratories entrusted me with this huge responsibility and provided me with guidance along the way.
As I am in the process of learning and becoming a better user experience researcher, I will be happy to receive feedback or suggestions to improve.
