Usability Testing for Augmented Reality
Takeaways from a rapid design process
Augmented Reality (AR) is disrupting a lot of industries these days. It is helping technicians maintain and repair equipment, allowing us to visualize the size and scale of things and helping us navigate our surroundings. The use cases are expanding by the day, as are the various different methods of interaction.
So, how do we build the next great application? What works and what doesn’t? These are questions circling the industry and while there are some fantastic guides to AR design out there, there are few proven methods for a solid AR design process.
One of the most challenging areas of interest in AR application design is how to test ideas. With a growing number of users starting to understand the technology from very different levels of knowledge, where do we start? And how do we get this technology in our users’ hands quickly enough to make an impact? These are a few of the many questions my team had when we recently tried to tackle AR usability testing, and while the process wasn’t 100% smooth, we came out the other side with some great insight into how to approach this issue more effectively moving forward.
Why is AR testing so hard?
Before I jump into the process, lets take a brief look at the challenges specific to Augmented Reality: Prototyping, user’s understanding of the technology and hardware fatigue.
1. How do we prototype quickly?
Augmented Reality, by its nature, is more difficult to prototype than typical software applications. Content will live both on a screen as well as in the real world. Users will be experiencing it in different environments and will likely need to move around in order to fully realize value. Because of all of these, it can be tough to rapidly create prototypes that represent the application in use without being too far from the actual experience to be useful.
2. Users Are Unfamiliar With AR
Augmented Reality introduces some new considerations for users that many are not used to. For example:
- How to wear or use a device properly
- Moving around and looking in different directions
- Interacting with different types of content
3. AR Hardware Causes Physical Fatigue
Augmented Reality provides a lot of freedom for content types all over the real world. It can be placed in relation to objects, rooms, around town, etc. Because of this, users are, and should be, encouraged to look and move around. This type of behavior isn’t new to most users, but doing so while wearing a head mounted display (HMD) or holding a tablet or smartphone for extended periods of time can cause discomfort and fatigue.
So, How Did We Do It?
So, how did we tackle the testing process given all of these challenges? Let’s jump into our project and examine the different phases of usability test development. First, we need to know what we are trying to accomplish.
The Project
For this project, we wanted to design and test some onboarding for Vuforia Expert Capture (VEC) over a very short timeframe, about a month. VEC is a tool used to allow a subject matter expert (SME) to capture a process via a head-mounted display (HMD) in step-by-step instructions. These can be captured via both the Realwear and Hololens; we focused on Hololens for this project.
The instructions generated by VEC can be used by anyone who has access in order to add contextualized details while they carry out the captured procedure.
For this project, we had two main goals:
- Determine if onboarding itself makes an impact
- Identify which style of onboarding most affected user understanding of the product
If you are interested in our overall design, prototype, and test process breakdown, you can read more about the project here.
Step 1: Design For Quality, Prototype For Speed
We used 2D tools like Sketch, Invision and Photoshop to build designs out quickly. We also used 3D tools such as Sketchbox and Unity to examine how the onboarding could look through a Hololens and prove out the feasibility of implementation and feel of the application.
This allowed us to design quickly in a professional manner without needing to build it into the actual product. If you are interested in learning more about these prototyping tools and how they are used for AR, my colleague breaks them down here.
After building some designs and verifying their feasibility, we then shifted to quickly making them consumable for test participants. In order to speed this process up, we adapted our designs to fit a tool we could use for rapid implementation, Powerpoint.
We chose powerpoint for a few reasons.
- Speed of implementation
- Fit with a tutorial based approach
- Interactivity for the self directed onboarding concept (UI Callouts)
- User Familiarity with using the software
What Did We Prototype
While we designed five different styles of onboarding with varying levels of complexity and potential benefit, we decided to focus on the three in the center, i.e. we skipped the easiest and the hardest to implement. We did so based on research into onboarding and instructional design as well as the number of available test participants we had available for this round of testing. The goal was to have at least 3–5 participants for each test in order to avoid individual biases while still getting a feel for effectiveness.
After selecting these 3 categories, we set to work building them into powerpoint. For the UI Callouts with explainers (Option 2)and UI walkthrough (Option 3) we built two separate powerpoint decks. The first being an interactive clickable UI mock-up, and the second being a standard step-by-step presentation. For the Active Walkthrough (option 4), we used the same presentation we built for the walkthrough, but had the user wear the device at the same time and actually perform each action before we allowed them to move on to the next step in the onboarding experience.
Step 2: Find and Prepare Test Participants
Once we had a prototype built, we needed to find participants. Since we wanted to get early stage tests done quickly, we reached out to our local network of coworkers. While it would have been more ideal to test with actual users, we decided that our local network would be sufficient given the speed and early stage of the project.
In the end we were able to get 12 different participants with varying degrees of technical experience. None of them had used a hololens or Vuforia Expert Capture before.
Our first step was to introduce them to the hardware. Luckily, Hololens has a great tutorial called ”gestures” which walks users through how to use the device. We had each user go through this tutorial before performing our test.
This was a point where it is worth mentioning the effort to avoid fatigue. Once the user completed the tutorial, we had them remove the headset while we explained how the test would work. This reduced the time the device was on their head, and therefore some of the tension associated with discomfort was removed from the testing process.
Step 3: Perform Usability Tests
In order to gain value from the test itself, we wanted to measure a few different aspects of the user’s experience. Our first goal was to determine if onboarding of any kind made a difference in the user experience. We did this by separating out one group of participants to do a control test. This group carried out our primary task without any onboarding.
The second goal was to determine a style of onboarding that was most effective. This was measured by splitting the remaining participants into 3 groups in order to test out each of our prototypes.
The test was performed by following the same format for each user:
- If the test was not a control test, the user was given one of 3 onboarding experiences. Each used a powerpoint prototype and one group used the hololens to perform tasks throughout this process
- We had participants in all 4 groups perform the same task (more on this next). While they performed this task, we took observational notes and asked the user to think out loud
- After completing the task, we administered a survey to better understand how the participants felt about application and their understanding of how to use it
The Onboarding
Onboarding was completed using powerpoint slides following one of our defined styles. For the first two groups, we simply had them run through the onboarding material at their own pace, then tell us when they were ready. The third group wore the headset while going through the onboarding experience and we did not allow them to continue on to the next steps until they had actually performed a function in the Hololens. Once participants completed this process, we had them remove the Hololens while describing the task to them. This was done in order to avoid extra hardware fatigue that was unrelated to the applications use.
The Task
Vuforia Expert Capture is used in the industry to help organizations digitize their procedures. Since this is commonly used for enterprise standard operating procedures (SOPs), we wanted to find a SOP of our own that our participants might be familiar with. Because of this, we decided to have our participants capture the process of using one of the lockers in PTC’s main office. This process was written out in steps on a piece of paper and was routinely performed by the participants we were testing, so it served as an ideal replacement for an enterprise style SOP.
The Surveys
The surveys focused on the participants comfort level as well as some open ended questions to see if the participants could envision ways in which they would use various features of the application. This allowed us to better understand what the confidence levels were between different forms of onboarding as compared to the control group.
Our Takeaways From AR Usability Testing
After completing the testing phase, we circled back and spent some time gathering our thoughts about the process. We gained some interesting insights into testing for AR in general and expect to be implementing things more effectively in the future because of them.
Since research around AR user testing is slim, we wanted to share our takeaways with anyone looking to perform a similar testing process in the hopes of driving the industry towards better testing practices. Here are the biggest takeaways we had from our experience:
Speed Over Quality
Like all usability testing, getting things in your user’s hands quickly matters. The earlier you can find flaws in your design, the better. In the case of AR, this means accepting that there will be some intermediate steps you may test. For us, this was using powerpoint rather than in-application tools. While this is nowhere near the end goal of what onboarding could be, it gave us some great insight into our designs so that future testing will be better informed. This saves time and resources and lets you get closer to the best solution by the time you actually build it out.
Behavior Vs. Attitude
Because of the technical nature of this test, we found a wide variety of confidence levels. Sometimes this was related to using technology the user was not used to, other times because of the cost of the equipment and the fear of breaking it. Whatever it was, attitude differed wildly between participants. We learned very quickly that focusing on what people did and said was more valuable to use than how they claimed they felt. While this was still useful for understanding how confidence levels were affected by our onboarding, we would need to do a better job of building a baseline for each user in the future in order to gain more value from this information.
Unknown Hardware Results in Unexpected Issues
While we were focused on testing the application’s onboarding, we certainly found some hardware related difficulties. Sometimes the device froze, or behaved differently than the participants expected. Many people used it with a square body, i.e. not turning their head, but turning their whole body instead. It was also uncomfortable and resulted in participants consistently stopping to adjust. Things like this may not always be taken into account when designing applications, and it is important to understand how a user uses the hardware itself and what problems this can cause. While the application may not be hard for the user to grasp, putting it into practice may offer new and unexpected issues.
Be Open to New Discoveries
When performing usability testing, it’s easy to lose sight of valuable information beyond what you are specifically testing. There is plenty of insight into how users interact with the application that may be valuable to recognize. For example, we noticed that users had difficulty framing their captures because of the difference between the Hololens field of view and what the camera captures.
Get Out There and Do It
Augmented Reality can be intimidating to many people. The technology, hardware and concepts might not seem immediately approachable. But that’s because it is new and exciting. There is so much more to learn about implementation and testing out there that has yet to be discovered. It doesn’t take an expert to design and run tests with low fidelity prototypes, and while you may not create the best possible scenario, you will most certainly learn something new. So get out there and do it. Design something, test it and gather your insights. We, as an industry, are all there with you and we have so much more to learn!
