Tips to test beta apps remotely

A collection of learnings and insights from a recent round of usability testing (August 2020)

Prototypes and working software (including Beta versions) are some of the most commonly used assets we (UX designers/researchers) use to validate our ideas, assumptions and hypotheses when designing and building digital products.

At Transpire and before the coronavirus pandemic, we would invite participants to our office to test and provide feedback on beta apps installed on our test devices.

In this scenario, we had full control over the process and could intervene easily if something went wrong. But ever since COVID-19 knocked on our door and said, ‘no one-on-one interviews’, we have had to adapt our approach to conducting remote usability testing.

The biggest challenge has been finding a way to test beta apps remotely, as we can’t provide users with devices — they must use their own.

I’m sure there may be tons of different ways to tackle this issue. But I wanted to share with you the steps we took, as well as the tools we used, on a recent project.

Imagine this to be like a cooking recipe, I’m going to enlist the ‘ingredients’ or tools we used (Google Suite products), but mentioning some alternatives in case you want to interchange it with your own suite.

Google Meet app (iOS and Android)
This tool will help the participant join the video conference and share their phone/tablet device screen.

Google Meet (PC or Mac)
Alternate ingredient: You can replace it with Zoom, MS Teams, Webex, or your preferred video conferencing software.
This video conferencing tool will help you record and store the session (straight into your Google Drive). In addition, it will allow you (with permission) to capture your participants’ face, gestures and body language.

Google Forms (Browser-based)
Alternate ingredient: You can replace it with Microsoft Forms, Typeform, or your preferred Form Builder app/software.
This tool will allow your participants to read and sign any documents prior to the session. e.g., consent forms, plain language statements, etc.

Google Docs (Browser-based)
Alternate ingredient: You can replace it with Microsoft Word, Apple Notes or your preferred word processing software.
This tool will allow you to write your script/discussion guide, screener, hypotheses, etc. with other team members and stakeholders.

Google Calendar (Browser-based)
Alternate ingredient: You can replace it with Microsoft Outlook, Apple Calendar or any calendar app.
This app will help you better organise and plan your sessions ahead.

Beta testing accounts (TestFlight for iOS, and Play Store for Android users)
You’ll need to create testing accounts to let participants download and install your beta app. You will find more information about this further down.

Trello (Browser, Mac or PC)
Alternative ingredient: You can replace it with Microsoft Word, Apple Notes or your preferred note-taking software.
‘What! You use Trello to take notes?’ We sure do! We use it to speed up our note-taking, analysis and synthesis. Want to see it in action? Get in touch with us! info@transpire.com.

The setup

Interviewer (You… duh!)

An illustration of a researcher holding a notebook

Ingredients

  • Google Calendar (to keep track of the sessions’ order)
  • Google Meet (to join the session using a computer)
  • Google Docs (to follow the structure of the discussion guide)
  • Google Forms (to make sure the participants read and sign any required form prior to the session)

Note-taker (Can be a colleague or even better… a stakeholder)

An illustration of a woman taking notes on her laptop
An illustration of a woman taking notes on her laptop

Ingredients

  • Google Meet (to join the session using a computer)
  • Trello (to take notes)
  • Google Docs (to follow the structure of the discussion guide)

Interviewee (The participant)

An illustration of a female participant joining a video call using her phone and her laptop
An illustration of a female participant joining a video call using her phone and her laptop

Ingredients

  • Testflight or Play Store email invite (to download and install the beta app in their personal device)
  • Google Meet app (to join the session and to share their phone/tablet screen)
  • Google Meet (to join the session using a computer)

In order to make this ‘recipe’ a total success (and also to make all the information a tad more ‘digestible’ and easier to read), I’ve chosen to split the information of this post into what I consider the five different stages of testing.

1. A week prior to the session

2. Two to three days prior to the session

3. A day prior to the session

4. The day of the session

5. During the session

An illustration of a woman smiling holding a notebook with her right hand and waving her left hand.

1. A week prior to the session

If this is the first time you conduct a session like this (i.e. remote usability testing a beta app), I would recommend you allow for extra time (maybe two weeks) as there’s a fair amount of prep to do.

Step one

Create a document for your discussion guide and research brief/goals

Ingredient: Google Docs

This document should include:

  • a list of your initial hypotheses and assumptions.
  • your screener questions; intended to identify your target users and weed out those who aren’t suitable for the study.
  • the discussion guide… in other words, the script and structure you are planning to base your conversation/interview from.

We would recommend sharing this document as well with your note-taker(s) and the client (getting them involved early in these types of activities will help you build rapport, give research an added layer of credibility and help them find answers to specific questions they may have).

Step two

Recruit participants

There are some good recruitment options to choose from when looking for participants for research.

Most of the time, we tend to use Farron Research or Askable. I personally prefer Askable as I feel it’s faster to set up, have more control in selecting the right participants and contact support (if needed).

Farron research and Askable logos slightly overlapping each other
Farron research and Askable logos slightly overlapping each other

Step three

Set the participants’ expectations as part of a consent form

Ingredient: Google Forms

Tell them:

  • you are looking forward to chatting with them (it’s always nice to feel welcome).
  • the reason why you are conducting the research.
  • what the session is about.
  • about the risks and/or benefits of taking part.
  • the tasks you will ask them to perform (at a high level).
  • the apps/software they will need to download.
  • how the collected data (audio/video) is going to be managed.
  • who to talk to if you are not handling the incentives.
  • read and sign the form if they agree with it.

We usually send to our participants one Google Form including both the Plain Language Statement and Consent Form.

Step four

Set up the beta testing accounts

Ingredient: Beta testing accounts

This is a step that you most probably won’t be able to do on your own.

Ask your Dev/QA team to help you create the necessary beta testing accounts (iOS=TestFlight, Android= Google Play Store) for your sessions.

You will only need to provide them with three main things:

1. the participant’s email address.

2. specify if they are Android or iOS users.

3. the date the beta app will be available for them to download. We recommend activating the beta accounts of each of the participants a day or some hours prior to their session to ensure they don’t get to ‘play’ with it and skew the results.

Important: If you are testing an Android app, it’s important to mention that the Play Store only allows participants with Gmail accounts to be invited to the testing program. If any of your participants have a different email provider, you will need to ask them to create a Gmail account. On the other hand, TestFlight (iOS) is more flexible and allows you to invite participants to their testing program regardless of the email provider they use.

Step five

Reduce the likelihood of having a ‘choppy’ call by verifying your participants’ Internet speed*

The participants will be joining the meeting using their phone/tablet (Google Meet App) and their computer (Google Meet). This may push their Internet speed limits.

Ask them to:

  • perform an Internet speed test from where they envision to join the session**.
  • send you a screenshot of their results (this can help you re-evaluate if a participant is suitable for the study or not).

I personally recommend using Speedtest.

*We’ve learned that if both the download/upload speeds are less than 10Mbps, the likelihood of having connectivity issues is high.

*Participants may have different Internet speeds at home, at work or when tethering their mobile device to their computer.

Step six

Create and send calendar invites for each of the upcoming sessions

Ingredient: Google Calendar

For the sake of keeping everything organised, giving you peace of mind and providing your participants, note-taker(s) and the client visibility and clarity on when the sessions are happening, we recommend creating a calendar invite for each of them.

Make sure you are inviting:

  • the participant
  • the note-taker
  • potential observers (usually client-side)

Important: In order to comply with the General Data Protection Regulation “GDPR”, we recommend to hide the guest list when creating the calendar invite as this will better protect your participants’ information.

Step seven

Create and set up your note-taking document

We would recommend to:

  • name your document including the number of participants you are going to talk to, the sprint number (if you work with scrum) and the date.
  • write some very basic instructions on how to use your preferred note-taking tool or shortcuts to edit, delete, etc.
  • use a separate page/section per participant.
  • include a screenshot of your participant’s Internet speed test (this can help you flag that a participant may have connectivity issues during the session).
  • share this document with your note-taker(s).
An illustration of a smiling woman wearing a red shirt pointing up with her left hand

2. Two to three days prior to the session

Step eight

Send all of the participants an email further explaining what the day of the session is going to be like

Tell them:

  • they will need to join the session using both their phone and their computer.
  • to download and install the Google Meet app as it will help them join the session through their phone and share their screen.
  • they will receive an email prior to the session to download the beta app.
  • to join the session using their computer (accessing via the calendar invite you previously sent them… refer to step four) 10–15 minutes earlier to ensure everything is working properly.
  • to contact you if they have any questions.
An illustration of a smiling man wearing a red shirt and drinking coffee with his left hand

3. A day before the session

Step nine

Make sure the participants have downloaded and installed the beta app

It won’t hurt if you contact them directly and check if they had any issues when following the instructions of each of the app stores. Two of the iOS participants from our last session assumed they had already installed the beta app when in reality they only installed the TestFlight app on their phones.

  • iOS participants will receive an email inviting them to download the TestFlight app. Once installed, this app will display an option to install the beta app you’re testing.
  • Android participants will receive an email with a link that will let them download and install the beta app directly from the Google Play Store.

Step ten

Remind them to join the session using their phone and their computer

Some of the participants that we talked to in our last round of testing thought they only needed to join the session using their phones. Make sure they understand they will need to join the session using their computer as well.

Double-check they have downloaded and installed the Google Meet app in their phones and remind them to join the session using their computers as well (we recommend to use the Google Chrome browser*).

  • Using products that belong to the same family of products (in this case Google Suite) reduces the likelihood of something going wrong.
An illustration of a smiling woman using her mobile phone with her right hand

4. The day of the session

Ensure everything is working. You want to get the most out of your session.

Step eleven

Send the participants a reminder to join the session 10–15 minutes earlier to make sure everything is working properly

There will be some variables out of your control. Ask your participants to join a tad earlier to avoid surprises. E.g. making sure their cameras and microphones are connected, helping/guiding them on how to join the session through their phones, etc.

There’s a chance you’ll have to act as tech support alongside your UX duties. Just remember to be patient with participants and provide guidance throughout.

Step twelve

Have your participants’ phone numbers and emails handy

Eight out of the last ten participants we interviewed, arrived late even though we reminded them a couple of times (through different means) to show up earlier.

Have their details ready… just in case.

An illustration of a researcher conducting a remote user testing session

5. During the session

Everything should be running smoothly. However, be prepared in case something doesn’t work as planned.

Step thirteen

Beware of the echo

There’s a high chance that you’ll experience some echo during the session as the participants will join the session using both their phone/tablet and their computer.

In order to reduce the chances ask the participant:

  • to mute the microphone on their phone and reduce the volume to 50 percent or less. Otherwise, this might cause a horrendous noise/ echo (also known as audio feedback) that might “blow your ears out”… and you don’t want that…believe me.
  • not to get their phone/tablet too close to their computer (especially to the microphone or the speakers' area).
  • use a headset or headphones on the computer.

Step fourteen

You might face some challenges when sharing your screen

Ingredient: Google Meet app

We often need to share supportive material with our participants (eg., images, design assets, etc.). To achieve this remotely, you most probably will need to share your screen.

Google Meet recently released a feature that allows multiple users to share their screens simultaneously. However, we found out that this feature only works well if all the invitees join the session using their computers. It doesn’t work as expected if someone joins and shares their screen using the Google Meet app (on their phone/tablet).

If you (researcher) share your screen (using your computer) and take over the participant’s presentation (phone screen share), the Google Meet app will automatically ‘go to sleep’ and stop sharing. The user will need to re-share their phone screen to proceed with the test(s). It might not sound like a lot, but it’s painful.

An illustration of a group of people having a chat

Hopefully, I’ve shown you the complexities of conducting remote user testing with non-live (test apps) on participants’ own devices.

Happy remote testing!

If you have any feedback or thoughts feel free to add them into the comments section.

All images were generated in Sketch using an amazing illustration library plug-in called ‘Blush’. The name of the artist/illustrator is Vijay Verma. You should check his work! It’s incredible!

I love creating positive-impact products that everyone can use. In other words, design products that are universal, accessible and inclusive.