Prototyping methods for augmented reality

The augmented reality (AR) industry is experiencing incredible growth. Investors are pouring billions of dollars into technology and talent. Hardware and software revenue is slated to reach $100B in the next couple of years, and we are seeing major strides in device (e.g. HoloLens) and software (e.g. Vuforia) user experience. It’s awesome.

Now in order to keep this momentum going, there are all sorts of challenges to overcome. One critical area is designing prototypes that envision how the end user is going to get value. There are plenty of tools and methods to do this for traditional desktop and mobile design, but that’s not the case in the AR space.

In this article, I’ll share with you some frameworks, methods and tools our team has used to tackle AR prototyping. To see how these approaches work in a project, I’ll take you through a recent project that focused on designing AR onboarding concepts for Vuforia and HoloLens. We had a short time frame, about a month to complete the project. In that period we did background research (1 week), created learning content (2–3 days), built prototypes (2 weeks), tested solutions (1 week), and presented our results (1–2 days). Let’s take a closer look at the activities starting with our design process.

Design Process

To coordinate our workflow, we used a modified Build-Measure-Learn process. We started with Learn and then stepped through each phase with a focus on AR prototyping needs. This model takes you from high level market research down to user testing of specific UI features.

1. Learn — get oriented and inspired

Starting with Learn is a great way to get oriented and find inspiration. Activities in this phase include reviewing and testing technologies, reading industry reports, and interviewing subject-matter experts (SMEs).

Product Testing

In the AR onboarding project we reviewed AR apps, which included Vuforia as well as similar apps (e.g. Dynamics 365 Guides) and apps from different markets (e.g. Fragments). We also looked at exemplar software in other industries such as Adobe and Unity. In each case, we paid attention to how they onboarded their users and took note of differences between AR and traditional software experiences.

For example, some good AR designs included:

• Encouraging spatial interactions: Prompt users to move and look around. For example, on some HoloLens apps, you have to map out your space with a blue mesh. Not only does it look cool but it also shows the user their interaction area.
• Multi-sensory experiences: Sound, touch, and visuals are all affordances of augmented reality; there is even some pioneering work on bringing smell to AR. This toolkit of sensory inputs provides multiple ways to alert, inform, guide, and engage users.
• Right content for the job: in the AR space there are design patterns emerging around user interactions. For example, reading text is best suited for on-screen layouts, not positioned in 3D space. Whereas showing animations and 3D holograms is great for in-scene experiences such as showing someone how to fix a machine by overlaying 3D content.

and some bad AR designs included:

• Content misplacement: There were several AR apps where key information was either not in the field of view (FOV) or oriented the wrong way (e.g. not having billboard activated)
• Information overload: In many cases there was too much stimulus. This included too much text on an info panel or too many types of content presented at one time such as audio background, 2D info panels, and 3D model animations.
• Skipping hardware orientation: Many users don’t know how to use AR equipment. Putting it on, adjusting the glasses and using the right gestures are foreign to most users. Most apps incorrectly assumed the user knew these fundamentals.

Industry Reports

Along with product testing, we also looked at industry research related to onboarding. This took us down the rabbit hole of how people learn, covering things like Bloom’s Taxonomy, Understanding by Design, ADDIE, etc. We also tapped into the design community and learned about the principles of onboarding. All of these resources helped us frame our thinking around what is important in a user experience.

For example, here are some general principles to keep in mind for onboarding:

• Show the happy path: Show users the best work flows so they learn best practices and see the breadth of the product’s value
• Track progress: As users progress through onboarding, add progress tracking so they feel a sense of reward. Tip: start the progress tracking with some check marks already completed to evoke a sense of investment.
• Explain the why: Help users understand why they are learning x,y,z features or specific workflows; this helps them understand the value of the tool and how it enables them to do their job better.

If you want a deeper dive on AR onboarding principles, check out this article from Jonathan, one of the designers on our team.

Subject Matter Expert (SME) Interviews

Getting insights from experts is another important part of the Learn phase. Common activities are interviews, email exchanges, and informal conversations.

In the AR onboarding project, our team consisted of SMEs on relevant topics (Vuforia, Work Instructions). As a result, we were able to draw on our own expertise in these key areas throughout the project, rather than needing to see outside experts. However, with many projects this isn’t always the case. To see what this looks like when you don’t have SMEs on the team, let’s take a quick look at a design sprint we ran for a manufacturing demo.

In this one-week project, we interviewed the builders of the manufacturing equipment and they helped us pin point key problem areas (e.g. up time) and potential AR solutions. What we learned is that the puck (a tray that travels along the track of the assembly line) would fatigue, and required service to keep the line up and running. Since the assembly had accident-prone areas, requires initial diagnostics, and included detailed steps, we chose to target this area for AR work instructions.

Alight, let’s jump to the next stage, Build, where we created the AR content and prototypes.

2. Build — bring your ideas to life

The Build stage is where you bring ideas to life. Ideating, creating visual mockups, and designing prototypes are common activities. In the AR onboarding project, we focused on two deliverables. The first was the instructional design of the content. The second was the actual AR prototypes.

Content Development

Our research in the Learn stage informed our thinking on the first deliverable. To kick start this effort we had a one-day workshop where we brainstormed, outlined, and refined different types of onboarding. We ended up creating a framework to organize this content, which started with a basic onboarding experience to full on project-based learning (see table below).

From an instructional design perspective, levels 1 and 2 are non-linear learning experiences focused on reference info, whereas levels 2, 3, and 4 are linear learning experiences focused on step-by-step instructions. Put more simply, levels 1 and 2 represented documentation-type instruction, and levels 3–5 represented tutorial-type instruction.

With these onboarding options fleshed out, we built AR prototypes of each one. We used a variety of tools to do this and, in some cases, had to invent new tool paths to build the mockups.

2D Prototyping

For preliminary concepts, 2D design tools work just fine. One approach is to envision the final solution with a 2D illustration. For example, here is a concept from another project that envisioned how a service technician would get AR guidance.

Another option is using 2D design tools to create icons, buttons, and other 2D features you want in the AR experience. For example, we used the Adobe suite to create mockups of the level 1 onboarding experience. We called this the “game controller” experience because it looks a lot like what video games show when you want to see the controller functions.

You can also create clickable prototypes with 2D tools. One low fidelity approach is using presentation tools like Keynote or PowerPoint. Many people know how to use these tools, so what is nice is you don’t have to have professional designer skills and tools. Here are the basics steps to do this:

1. Take a picture of the setting where you want to place an AR experience (e.g. room, office, factory work cell, etc.), then paste this image in PPT. You can also use screen shots of a 3D modeled environment like the image above.
2. Add persistent UI elements such as Nav menus and branding; this is your template slide. What’s nice about this approach is adding basic shapes (boxes, circles), color and text is typically all you need to do.
3. Copy and paste the template slide, and then design specific interactions such as selecting a view or initiating an interaction. When you are done, you can go into presentation mode and see the animated experience.

You can also take your PPT prototypes to a higher level. For example, in the onboarding project we took a picture of the setting, and then we used Sketch to make professional level UI elements. To add logic we used the hyperlink function in PPT.

3D Prototyping

The next level of prototyping is building 3D experiences. A common approach is using Unity and then consuming the experience in Vuforia. This approach is more advanced than other approaches I mentioned, as you need to know how to use Unity’s tool set and coding is required if you want to add logic. Once you get a hang of the tool, you can create prototypes within hours and check them out in the HoloLens, which is really helpful.

The high-level steps are:

1. Follow guides to download and install MRTK, and then set up your first HoloLens development environment
2. Populate Unity scenes with and 3D or 2D elements that you built out. Pro tip: group assets from similar scenes under parent game objects for easy visibility toggling.
3. Build out your scenes using Visual Studio for deployment on the HoloLens. Now, you can view your designs in AR on the HoloLens!

As a side note, there are some up and coming AR design tools that don’t require coding such as Torch which is a mobile app or wiarframe which is a desktop app.

Prototyping AR with VR

Another cool method for AR prototyping is using virtual reality (VR) tools. Our team has found that VR is really helpful for doing quick renders to get a sense of spatial layouts. For example, if you have an AR app that provides factory work instructions, you can use VR to quickly render where the 3D machine and 2D work instructions are placed, and then jump in and get an immersive experience.

The VR prototyping tool we used in this project was Sketchbox. The high level steps for this approach is:

1. Create the 2D UI elements. For the onboarding project we used Sketch to design the 2D info panels.
2. Add assets to the VR experience. You can add 2D assets that you create. An added bonus is VR tools like this typically have a 3D model library. For example, in our project we imported a 3D microscope from a Google Poly library. This definitely helps with speeding up the prototyping process.
3. Lay out the scene. What is great here is that design and validation happen concurrently. As you layout the scene, you see the layout in VR, and when you need to move things around its dynamic. No waiting time for publishing.

As a side note, we also explored using Sketchbox to create envisioned future concepts. A traditional approach to this technique is creating 2D illustrations like the AR service guidance graphic above. With VR you can take it to the next level by merging VR images with real world photos.

For example, in the sequence below we took a screen shot of the VR setup, then took a photo of someone using the HoloLens, then wiped out the VR background, and it merged the two images to give you a sense of the future experience. What’s great about this approach is the quick turnaround time. It took us 30 minutes to make this mockup!

If you are curious about these prototyping tools, Dylan wrote an article that goes deeper into this topic. Ok, now to the final stage, Test, where you get insights on your prototypes.

3. Test — see what works and doesn’t

In the Test stage you find out what works, and what doesn’t. The goal is to get insights that will inform future stages of design and development. At a high level a user test looks something like this

1. Design the test
2. Recruit participants
3. Implement the test
4. Analyze results
5. Document findings

If you are curious about these general practices and how they apply to the UX field, check out this overview. For this section, I’ll focus on unique aspects from doing AR projects. Before jumping into that, here is a little context so you can see how this played out for the AR onboarding project.

For the AR onboarding project we wanted to determine if the prototypes impacted both attitude and behavior. To to do this we set up a control-based experiment. For the control, the end user would learn how to use HoloLens gestures and then would get on Vuforia to complete a set of work instructions; no onboarding was given. The experimental group did all the things the control group did as well as consumed the onboarding experience.

We measured behavior by observing users actions and interviewing them at the end of the study. To measure confidence levels, we administered a survey. In addition, we also measured if different levels of onboarding had different outcomes. We had three experimental groups (level 2, 3 and 4). If you are curious to learn more about this test design, check out this article written by Dan who dives in the test stage of this project.

Low Fidelity User Testing — PPT + HoloLens

In the spirit of moving fast and not investing a ton of code into software, we created a low fidelity model for user testing on HoloLens.

There were two parts to this test model. One part was the participant consuming onboarding content through PowerPoint. The other part was interacting with Vuforia on the HoloLens app.

For example, in the level 4 study (active UI walkthrough), each participant wore the HoloLens while they looked at an instructional step on PowerPoint, and then they implemented that instruction (e.g. take a photo) on the Vuforia app. They repeated this process for each step. Its a quick way to emulate viewing 2D panels in AR using PowerPoint slides.

AR Hardware

When running tests in AR projects, there are a few unique considerations. Since many people are not familiar with AR experiences, several issues pop up:

• Comfort of the device (weight, form factor, heat)
• Using interactions (how to interact with the software from voice to gestures)
• Managing software on the device (installation, loading software, updates)
• Dealing with a new experience (immersive content, FOV affordances, spatial interactions)

To address some of these points, we had each participant (control and experimental) go through the HoloLens gestures experience. This familiarized them with the interaction possibilities and warmed them up to what an AR experience is like.

Onboarding results

As a side note, you may be curious to know how the results turned out for the onboarding project. The major finding was that onboarding did improve user confidence compared to the control, with increased confidence correlating to higher levels of onboarding.

In addition, we discovered that users engaged in more features in the AR app when they engaged in higher level onboarding experiences. For example, the control group didn’t use certain features (e.g. location) when they carried out the work instruction exercise.

Final thought — making the best use of your time

One of the most valuable resources a company has is time. Releasing products on time, delivering services in a timely manner, allocating staff time; all hit the bottom line. As such, finding ways to minimize time or use time effectively is critical. One lever a company has is using proven methods that streamline work. This is a gap in the AR industry. Many companies don’t have access to frameworks, techniques and associated tool paths to quickly research, develop and release AR solutions.

Our team has been working to fill this gap. For example, our team built a workshop called an AR Design Sprint that accelerates early stages of product design by bringing key stakeholders (product owner, business lead, tech experts) into a 3 day workshop where the result is a product vision, tech requirements, and initial mockups of the solution.

In this article, we covered methods for prototyping AR solutions rapidly. This includes a framework to coordinate team projects (Learn-Build-Test), key areas to focus your research on (industry reports, testing competitor/analog apps, SME interviews), hands-on techniques for low fidelity prototypes (2D, 3D, VR/AR), and testing considerations (low fidelity testing, hardware needs).

If you want to explore these topics further, my teammates wrote articles about onboarding design, tools for prototyping AR, and running user tests.