3 things I learned from conducting & analyzing remote user tests

Lesson 1: Zoom is your imperfect best friend

A first challenge I faced was to observe and record the participants’ screen while they were using the app on their own phone. I first thought of weird hacks to accomplish this with a laptop webcam, but hey, technology advanced!

  • It records of the participant’s shared screen in a sensible format
  • It’s possible for the participant to join with their phone screen and with their laptop webcam at the same time (why? see below)
  • The file size of a recorded video is remarkably low given the decent video quality: ~100 MB for 30 minutes.
  • You get the audio separately as well, which can be useful for sharing with AI transcription tools (see later)
Snapshot from a user test. Only my webcam is active.
  • On iOS the screen-sharing process is not that obvious. See these instructions.
  • On Android at least, the support for sharing both a screen feed and webcam feed from a phone is limited. It only works together when you’re in the Zoom app. That’s why you might want to use the laptop and phone at the same time.
“Honestly, I also don’t know what just happened”. We’re both laughing at a bug in the app.

Lesson 2: Be extra patient

Conducting a user test often means keeping a balance between interacting with the user and observing their natural exploration of a product, because both can give interesting results. For the users to act according to their normal behavior they need to feel at ease. To achieve that, your intrusions as interviewer should be minimal. But at the same time, you want to intrude. You want to ask questions and gain knowledge.

The lesson: bring in pauses and leave the participant ample time to think

Lesson 3: Focus on insights, not transcription

When live notes fail

The goal of user testing and interviewing is to extract actionable insights for product improvement. It is a keystone in the user-centered design process. But while doing remote user tests on your own, it’s easy to miss or forget interesting remarks or actions from the user.

Structured video analysis

A structured way to extract insights from recorded interviews, in contrast to live notes, is the process of coding: to assign “codes” to a transcript. That means tagging interesting bits so you can count their occurrences and compare them across interviews.

Using Atlas TI for video coding

Work setup of coding remote user test in Atlas TI with a transcript loaded. A big screen helps.
The video coding feature of Atlas TI is good!

Lesson 3: you don’t need a transcript to use a video-coding tool

However, you may consider using Otter.ai or something similar to generate a transcript anyway, it might help speed up the video coding slightly.

Some more tips

  • Send a pre-questionnaire with contextual and demographic questions. What prior experience does the participants have? It helps you prepare the interview. I used a Google Form now, but you might as well check out Typeform.
  • Use a booking tool to schedule sessions with participants. I used Calendly. Their free tier allows for one event type which sufficed. They offer a Zoom meeting integration for free until June now, but you can also post your Zoom link in the event description otherwise.
  • Have a small chat in the beginning of a session. We all like a chat now!
  • For richer information in mobile screen shares, you can show the taps from participants on Android phones. Check this article to see how (it involves Developer Settings). On iOS this feature is unfortunately not as readily available.
  • Aalto University-registered student? There is a university license for home use of Atlas TI. Check download.aalto.fi. Student somewhere else? Check your university’s software offering.
  • Atlas TI is cool, but if you want to import your own transcripts from oTrancribe or Otter.ai it falls hopelessly short. Here are some notes on how I imported my oTranscribe & Otter.ai transcripts
    (Update April 9: I made scripts to do this, they work well now)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Thor Galle

Thor Galle

15 Followers

HCI grad student & consultant at Columbia Road. I talk code, design, data, and at great length. Read with me on Readup: https://readup.com/@thorgalle