Beautiful Augmented Reality Design Patterns

After getting our hands on a NASA HoloLens, the first thingwe did was explore which existing design patterns could support our own prototype design. I’ve always been a fan of game interfaces, and so I immersed myself in a really cool AR game called Fragments.

(This was 100% research and not at all an excuse to play video games for work)

How does the HoloLens interact with the real world?

Before we talk about UX interactions, let’s go through how the HoloLens maps reality from a high-level overview.

1. Spatial Mapping
The HoloLens constantly updates its real-world mapping through its depth sensor. When the HoloLens detects a user “pinch” (a HoloLens gesture that is exactly like clicking) at an empty space, it sends out a radar ping of what it’s mapped.

A saved spatial map of our office

Here, we can see that some things have changed since the last scan. For example, the geometric mesh hasn’t been updated to account for the blue chair.

2. Hand Gestures
The Hololens navigates with the cursor. The cursor is always in the center of the field of vision, so users must move their head and pinch at the item the cursor is currently on. This is exactly like moving your mouse around a screen and clicking.

The window accurately maps to your hand and arm movements

By pinching a window’s title bar, users can drag the window anywhere in space. Extending the arm allows users to place windows further or closer to their position.

3. Depth Mapping
When the cursor (or AR item) is over a realobject, it changes shape to fit the object’s surface. This is useful when users want to place AR objects on a real desk or chair.

The cursor changes shape according to the depth map

What are some great HoloLens design patterns?

1. Use tooltips to progressively disclose voice commands
The HoloLens includes system-wide voice commands (i.e. “select”, “close”), and context-specific commands (i.e. “show download folder”).

Hovering over a button displays the voice command tooltip

Instead of showing an entire list of what words will activate the HoloLens, the system hides the commands in tooltips. Users hover over a button to bring up the tooltips.

2. Support active/passive states
The HoloLens can detect when a hand is in its field of view (FOV). This allows specific toggle modes that depend on user behavior, such as watching TV.

The cursor only appears when the hand is in view

Here, the Hololens show/hides the cursor on a full-screen video clip.

3. Use the “follow along” design for a better Heads Up Display
Heads Up Displays (HUD) are great for keeping information handy in your FOV at all times, but can be really annoying when it obstructs your view. If you’ve ever worn glasses with a smudge on it, you know what I’m talking about.

A solution to this is a “follow along” design, where the important piece of information is kept in the view at all times, but trails along the edge of the FOV. This allows more freedom than a traditional HUD. Users can now lock a piece of information, yet still look over and around it.

HUD follows the user’s gaze, providing feedforward that the user can look around it

Here, the HoloLens displays the “follow along” capability for horizontal motion, but not vertical.

3b. (exception to the rule)
In the game Fragments, the head-locked information is used sparingly, and only when showing a small, 3 second prompt. I found this to work very well.

HUD done right: The prompt is small and only appears for a few seconds

4. Make buttons Big (Fitts’ Law)
Compared with computer mice, navigating with a head is much less precise. When designing for head-style navigation, make buttons large enough to reduce error.

Fitts’ Law is important with head navigation

Here in Fragments, tool selection is done with large circular buttons along with snap-to capabilities.

5. Minimize Typing
As mentioned with Fitts’ Law and small buttons, typing with your head is literally a pain the neck. Avoid too much clicking in the HoloLens.

Typing with the head is literally a pain in the neck

6. Hide Contextual Information in the environment
AR provides a new way to progressively disclose information. In Fragments, specific holograms can be brought up when interacting with larger pieces.

The selected part appears in isolation, showing name and data

This would be useful in a Maintenance/Repair context, where a technician could bring up relevant information with merely a pinch of the hand.

7. Direct User Attention with these Methods
In Fragments, the game provides multiple feedforward mechanisms to alert the user to changes in the AR environment.

• Particle Trails
  As shown below, little blue shards begin to crystallize from your position towards the direction of an important object.

Trails of blue particles pinpoint where the object is in the room

• Arrow Indicators
  Alternatively, arrow indications can also help users orient themselves in an AR environment. As shown below, a “Look Here” indicator helps the game tell users where to look.

The “Look Here” turns green and loses the arrow when users are in the right location

• Light Beams
  AR can also direct users to a specific location, as shown below:

Using a vertical beam of light tells the users exactly where to go