UX methods: Automated Remote Research
A second post in a series on UX methods
It’s 6 p.m. in a cold, wintery Moscow, Russia. Alexandra, a 21-year old college student is studying at home when she receives a notification on her phone — she has been recruited to complete a usability test on Amazon.com. After following the link to the site, Alexandra is prompted to complete various pre-determined tasks while thinking aloud. Within 30 minutes, she completes the set of tasks and receives $30 in compensation.
Halfway across the globe, it’s 9:00 a.m. on a rainy day in Seattle. Brendan, a user experience researcher for Amazon, receives a complete audio recording and screen capture of Alexandra’s test session. He doesn’t have time to open it though (at least not right now) — it’s only one of dozens received from users around the world, each a data point in a usability study of Amazon’s latest new feature.
The above scenario illustrates how automated remote research can be used in the real world. Automated remote research (alternatively, unmoderated remote research), is a user experience research technique used as part of the product design process. It involves using an online software program that records the screen and/or voice of test participants, as they complete a set of pre-determined tasks using your website or application. It allows for software designers to determine if users can use the product to easily accomplish relevant tasks.
In general, usability studies answer the question of whether or not a design is easy to use and intuitive for the user [1]. It usually follows something like this: a product or experience is put in front of a test participant. The participant is then asked by the moderator to complete a series of tasks while they are observed. The moderator listens to the participants as he/she “thinks aloud” during the process (see “Method 87 — Think Aloud Protocol” from Universal Methods of Design” by Martin and Hanington). However, in-person usability testing can have many drawbacks [2]; the tests are very time-consuming, as one facilitator running the study can take days just to do a few people, which also can be difficult to recruit. As such, these studies are conducted with small sample sizes, which may not be entirely representative of a diverse user base. Also, the testing is done in the context of an environment that is usually not representative of the real-world use of the product.
This is where remote usability testing comes in — tests are remote and anyone with internet access can click a link and complete a study. The tests can be done in parallel at any time, resulting in high time-efficiency. The remote testing software can even recruit participants automatically for you. It’s also time efficient because you don’t need to schedule anything, as the user can do it at their own convenience. As such, larger sample sizes can easily result without much extra work for each additional participant. This can reduce the margin of error, allow for more significant insights, and be more representative of a range of users [1]. Also, since users can do it wherever they want, they are more likely to test the product in the context of the actual environment they would be using it in. An example that highlights this benefit would be a fitness app to help a user work out in the gym. Testing the app while the user is actually in a gym is much more realistic than testing it in a controlled office setting, which can reveal insights that otherwise would’ve been missed. Typical questions that automated remote research studies can answer include the following:
- What percentage of participants able to successfully complete a task that involves using the product?
- How long does it take on average for users to complete the task?
- Do users have trouble completing the task? If so, where do they run into the issue?
- What is the most common flow or click path that users take in the interface to get to where they need to be in order to complete the task?
Notice that most of these questions are quantitative measures — the automated remote research method’s biggest strength is that by easily getting large sample sizes, numerical statistics can be easily generated and used to measure the user experience of a product. However, more qualitative insight on why a user behaves the way they do should not be ignored either — this can be accomplished by evaluating the video and audio recordings and seeing exactly how the user is interacting with the product, as well as using other research methods.
Automated remote research should not be confused with moderated remote research. In a moderated remote study, the moderator or facilitator is in a different location than the user, and is sharing screens with the participant in real-time. This is where you can see some cons to the unmoderated method — since a facilitator isn’t present when the participant completes the study, there isn’t any way to respond to comments or ask detailed follow-up questions specific to the test participant based on their experience with the tasks. The follow-up questions will have to be created pre-emptively and hopefully capture relevant information. Also, the facilitator can’t help the user if they get confused about a task and get sidetracked, or start to forget to think aloud. As such, an automated remote research study is best suited for the following use cases [1] [2] [3]:
- The tasks being tested are direct and specific, so users are unlikely to get off-track.
- A large sample size is needed because you need a high degree of confidence (E.g. a feature’s design is critical and will have a major impact.)
- You want to reach a broad, diverse range of users that are geographically dispersed and difficult to bring in for an in-person test.
- It’s necessary to test the natural use of the product in the context or environment of its real world usage.
- There’s limited time available to get actionable results
The cases that are not suited for automated remote studies [4]:
- When you want to have tight control over confidential test material.
- The users don’t have reliable or any internet access.
- The visibility of the user’s face, body, or eye movements is important to the study.
Method
The method for automated remote study can be broken down to 5 easy steps:
- Define the study goals
- Select the appropriate remote research tool
- Create clear tasks with defined success and failure measures
- Recruit customers (if needed) and send out the test with clear expectations
- Summarize and analyze results
Step 1: Study Goals
Determine the goals of the study before jumping into the study. Without well thought-out goals and requirements, the study design may lead to less actionable or relevant data [5]. Identify the overall research objective, as well as the research questions that need answering. For example, a research objective can be “Evaluate the effectiveness of a booking application”, while research questions can be “Do users understand the field labels?” and “Do the error message help users along the process?”
Step 2: Select appropriate tool
There are many remote research tools available, as seen by Figure 1. For our purposes, we are most interested in the bottom-left quadrant. Even in that quadrant alone there are also many automated tools that prompt users to perform a series of tasks and records what they do, and how they do it.
For automated remote studies, the tool should have screen and audio capture capabilities. Ideally, the tool should also have the ability to give pre-test and follow-up surveys to the user. This can be used to gain insight on the context of their use and capture more qualitative insights (E.g. “On a scale of 1–5, how easy was it for you to use the app?”). Some tools also provide services to automatically recruit participants for you, saving you time. Examples of tools include Loop11 and UserTesting.com for desktop, and Lookback for mobile applications or websites.
Step 3: Tasks
The tasks in the study should match up with the research goals and questions. It’s very important to get the tasks and the wording of the tasks exactly right for automated remote research, since you can’t be there while the user is going through the test. Participants should be provided enough detail to complete the tasks successfully, including any fake data they may need while interacting with the prototype (e.g. fake credit card information). The task shouldn’t include too much detail, however, as it may confuse the user or completely give away how to do it, which would defeat the purpose of the test. The tasks can also include context of use to provide a more realistic scenario to the user (e.g. “Imagine you are planning a vacation to Banff National Park…”). Each task should also have clearly defined success and failure measures to allow for easy analysis later on.
Act 4: Recruit and Send out the Test
As mentioned earlier, some tools automatically recruit participants for you to complete the study. Other times, it may be necessary to recruit your own users, especially if you have a specific target demographic. This can be achieved through e-mailing an existing mailing list of customers, or by intercepting customers that are using the product if it’s already live, and offering an incentive for completing the task. The test should also set clear expectations for the participants to prevent frustration or surprises during the test, which can lead to higher quality data and completion rates [5]. This can be done by letting them know things like the time commitment for the study, the compensation, number of tasks, device requirements (e.g. audio and video will be recorded), etc. Once the entire study is put together, a pilot should be done with another person on your team to ensure everything runs smoothly. Then, the test is sent out, and you wait for the results to come back!
Act 5: Analysis
After the study is complete, analyze both the quantitative data and qualitative data. The quantitative data may be analyzed automatically by your research tool — it may provide graphs showing how many users were able to successfully complete the tasks, or give you an average score for how high your participants rated their experience with a certain feature. The qualitative data can be taken by watching the video and audio of the participants’ test sessions, which can be done with other stakeholders on your team to fully understand the insights. Notes can be taken down on sticky notes and clustered to help form meaningful and organized conclusions [7] (see “Chapter 17 — Learn” in “Sprint”.)
Summary
In summary, automated remote research is a powerful tool that allows you to time-efficiently evaluate the usability of a product through quantitative measures (success rates, task complete time, etc.) and qualitative measures (video and audio recordings, etc.)
Case Study #1
The Design and Usability Center at Bentley University conducted a usability study on ordering groceries from Amazon.com using the automated remote usability testing method via UserZoom [8]. They had 100 participants that had an age range of 18–55. A pre-study survey indicated that the main reason users would consider buying groceries online is for convenience and time. This is valuable insight, as it indicates a goal for the usability of the product should be that the interface should be extremely easy to use and allow users to shop quickly. They had three tasks — for example, the second one was “You want to buy Skippy Peanut Butter with a discount you heard about on Amazon.com. Please write down the discount code.” This is a well-written task: it puts the user in a scenario with some context, and the task is very specific with a well-defined validation criterion (whether or not they get the correct discount code.) They also had questionnaire at the end of the study that asked the participants to rank things such as ease of navigation and product search on a Likert scale from 1 to 5, providing concrete insight on how much the users actually enjoyed using the product.
The second tasks’ results revealed that 47% of participants thought it was very difficult to locate the code, based on the post-task survey. As seen in Figure 2 below, a heat map was also automatically generated to easily see where users went to find the code and whether or not they were successful.
Overall, the study revealed that the usability of the site to perform the given tasks were poor — 53% of participants thought the navigation was very difficult, and 55% of participates said they wouldn’t purchase their groceries in the future from Amazon. Although this maybe be because some people would never buy their groceries online regardless of the retailer, it’s valuable to have as a benchmark on the next iteration. The main obvious issue was that the gourmet / grocery was built like the book store, which is not useful for users. The automated remote method was able to efficiently quantify how big the issues were and revealed user’s thoughts and frustrations.
Case Study #2
Because automated remote research provides a lot of sound, quantitative data, comparison studies between products can be easily done. For example, in “User Experience Magazine,” Tom Tullis did a comparison study on two websites that had information on the Apollo space program, one being the official NASA website, the other being the Wikipedia article. He had a total of 130 participants. Each participant answered questions related to the Apollo space program, answers to all of which can be found on both websites. Again, these tasks have clear defined success measures (either they get the answer correct or don’t). Users were randomly assigned to one of the websites to complete the questions, of which the set-up can be seen in figure 3. Another benefit of the automated remote research study can be seen here, where a large sample size would be needed in order to do this research method of randomly assigning two different products to compare them without having a result that is bias and has a large error margin.
The results indicated that test participants who used the Wikipedia article to answer the questions got significantly more tasks correct than users of the NASA website (71% correct vs. 58% correct), and were on average 23 seconds shorter at answering them. They also rated their experience of completing the tasks as significantly easier on a 5-point scale (3.1 vs. 2.6). The accuracy, time, and rating were combined into an index to provide an overall usability score for each task. As seen in figure 4, the users who had the Wikipedia article as a reference to complete the task had overall higher usability scores.
Again, this shows the power of the easily quantitative automated remote research study. The numbers were obtained and analyzed, and used to quickly compare between the two products. Beyond the numbers, the open-ended questions also helped identify usability issues that may indicate why the NASA website got lower scores. For example, an example of one the comments for the NASA website was “Search on this site is next to useless”. This indicates that the search function is something that users attempted to use to complete the tasks but were unable to use it, and thus needs improvement.
Case Study #3
Another example of automated remote research illustrates how it can be used not only for fully functioning websites, but also for lower fidelity prototypes. UserZoom built a wireframe for a website using Azure, and had participants add a pair of women’s shoes to their online shopping cart, as seen in figure 5. [10]
UserZoom was able to validate the success of the user by noting if they were able to reach the correct URL. As seen in figure 6, a clickstream flowchart was produced to show what percentage of users navigated to which parts of the website. 90% of users were able to successfully complete the task.
References
[1] S. Lanoue, “What is Remote Usability Testing? | UserTesting Blog”, UserTesting Blog, 2015. [Online]. Available: https://www.usertesting.com/blog/2015/11/30/what-is-remote-usability-testing/. [Accessed: 04- Jul- 2016].
[2] C. Gray, “How to Run an Unmoderated Remote Usability Test (URUT) -”, UX Mastery, 2015. [Online]. Available: http://uxmastery.com/how-to-run-an-unmoderated-remote-usability-test-urut/. [Accessed: 04- Jul- 2016].
[3] A. Schade, “Remote Usability Tests: Moderated and Unmoderated”, Nngroup.com, 2016. [Online]. Available: https://www.nngroup.com/articles/remote-usability-tests/. [Accessed: 04- Jul- 2016].
[4] “Remote User Research & Usability Methods | UX Magazine”, Uxmag.com, 2016. [Online]. Available: https://uxmag.com/articles/remote-user-research-usability-methods. [Accessed: 04- Jul- 2016].
[5] “12 Best Practices Remote Unmoderated Usability Testing”, UserZoom zooming in on the customer experience, 2014. [Online]. Available: http://www.userzoom.com/uxguide/12-best-practices-remote-unmoderated-usability-testing/. [Accessed: 04- Jul- 2016].
[6] B. Martin and B. Hanington, Universal methods of design. Beverly, MA: Rockport Publishers, 2012.
[7] J. Knapp, Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days. Simon and Schuster, 2016.
[8] “Lab + Online Usability Testing — UserZoom zooming in on the customer experience”, UserZoom zooming in on the customer experience, 2010. [Online]. Available: http://www.userzoom.com/casestudy/lab-online-usability-testing/. [Accessed: 04- Jul- 2016].
[9] T. Tullis, “Automated Usability Testing: A Case Study”, User Experience Magazine, no. 7, p. 3, 2008.
[10] “UserZoom | Remote Prototype Usabilty Testing Case Study”, UserZoom zooming in on the customer experience, 2014. [Online]. Available: http://www.userzoom.com/casestudy/remote-prototype-testing/. [Accessed: 04- Jul- 2016].
