What I Learned Making 5 ARKit Prototypes

Five Concepts for Designing Augmented Reality Apps

Imagine a future where everyone is wearing “AR glasses”, and digital content isn’t limited to screens. Would that future look like this?

Maybe. It raises some questions:

1. If we can display content next to the watch, does the watch need a screen?
2. Do we need a physical watch at all?
3. How would a user interact with the content? Voice commands? Hand gestures?

ARKit prototype of a future weather app.

Prototyping the Future

These are good questions to ask and help us solve future UX problems. By materializing an idea as a prototype, we can experience it firsthand and explore possible improvements.

I built five prototypes using image tracking in ARKit 2, and would like to share what I’ve learned. Hopefully the following concepts help you design better AR experiences.

Concept #1: Screen Space vs World Space

When designing phone-based AR experiences, we need to think about where we place visual elements—should they be placed “on the screen” or “in the world”?

Here’s my prototype of a potential future shopping experience, where physical packaging comes to life in AR.

The video of the rotating AirPods is placed in the world, fixed to the position of the box. It’s placed in world space. The buttons at the bottom are placed on the screen, unmoving even when the camera moves. They’re placed in screen space.

Generally, visual elements should be placed in world space when they act like real objects, and should be placed in screen space for easy reading or control.

The Measure app on iOS 12 does a great job of finding the right balance between screen space and world space. The measurements are always in screen space, rotated to match the lines, and can transition to a larger, expanded format when tapped.

Concept #2: Setup Costs

AR has the ability to bring static media to life. In this prototype, I added videos on top of a print newspaper to make it more engaging.

The experience feels magical—straight out of Harry Potter. 🧙‍♂️

However, it suffers from the high setup costs of current AR apps. In order for a user to have this experience with a newspaper, they need to switch contexts, turning their attention from the newspaper to their phone.

On top of this context switch, they need to:

1. Download the app.
2. Launch it.
3. Allow camera access.
4. Physically move to position the camera or initialize AR.

These added steps dictate that AR content should be as meaningful and useful as possible. A newspaper with a special section explaining a concept in AR would be more compelling than a passive layer over the reading experience, and therefore more likely to motivate users to overcome the high setup cost.

Concept #3: Real World Context

AR content can be more compelling when it’s directly augmenting what the user is already experiencing.

This prototype identifies key figures in historical artwork, allowing the viewer to learn more about the paintings.

By highlighting the key figures in world space, the user continues to view the painting while learning more about it. The user is already physically viewing the painting—the app enhances their current experience.

Instead of switching contexts between a museum brochure (or a museum brochure app) and the painting, AR delivers additional information seamlessly.

Concept #4: Interaction Models

In this prototype, the user is able to see into the past. Artwork in this public graffiti park is constantly covered up, but this app lets the user peel back layers of paint to reveal past artwork.

The user can navigate forward or backward in time by tapping on the left or right sides of the screen. While great for demo purposes, without a prompt, the user experience is difficult.

Here are a few alternative interactions:

1. Provide explicit controls in screen space similar to scrubbing through a video.
2. Use gestures to directly manipulate the position of the images in world space, swiping left and right between them.
3. Determine the image to show based on the relative camera angle, similar to a lenticular print.

The best option for an AR interaction model depends a lot on the user’s context. In this case, the ground is rough and uneven, which might make it difficult to move around while looking through the camera. Direct manipulation might not make sense if the images of the artwork don’t align precisely.

Because the experience varies so much on the particular use case, it’s important to prototype early and often to determine the best interaction model. Oftentimes you won’t know what works best until you try it.

Concept #5: The Future

Until AR headsets are widely adopted, most AR applications will be experienced via mobile devices. While there are a lot of current constraints, it can be valuable to think about future possibilities.

Here’s the prototype from the beginning of this post, allowing watch apps to display more information in AR.

This is not currently practical—holding a phone to look at another screen doesn’t work too well.

However, exploring one step further than current technology can lead to new ideas today. What if we could use the digital crown on an Apple Watch to control some other kind of AR experience, or use haptic feedback to provide additional information about the surrounding world?

The Power of Prototypes

My goal with these prototypes is to clearly show others the potential of augmented reality. AR has an incredible number of use cases beyond placing virtual IKEA furniture in your living room.

Prototypes are powerful because they inspire. They should feel like a glimpse into tomorrow. They should set an aspirational goal—a vision for the future.

Takeaways

1. Consider which elements live in screen space versus world space.
2. Make your content compelling—AR apps have additional setup costs.
3. Augment the user’s current real-world context.
4. Think through various interaction models—prototype whenever possible.
5. Imagine future possibilities—it will be here sooner than you think.

Additional Resources

If you want to learn more about AR design (and specifically designing ARKit apps), I highly recommend these resources:

1. WWDC18 Presentation “Creating Great AR Experiences”
2. WWDC18 Presentation “Prototyping for AR”
3. Apple’s Human Interface Guidelines

If you enjoyed this post, please leave some claps. 👏👏👏

You can clap up to 50 times, so get clicking/tapping! 😉

If you like this kind of stuff, you should follow me on Twitter. I only post high-quality tweets. twitter.com/nathangitter

Thanks to David Okun for revising drafts of this post.