Mixed Reality Experiences


By now you’ve heard about the Hololens — it’s essentially a computer that you wear on your head. Hololens wearers see the real world around them with virtual objects that float or are mapped to their space in the real world.

Roboraid for the Hololens is a first person shooter game where robots bust in through your walls and attack

The industrial design of the Hololens stretches to fit heads of all shapes and sizes, is surprisingly comfortable and beautifully weighted.

The Hololens hardware is designed to allow the user to pull the glasses up to the top of their head when they are not in use and pull back down while in use. I had the Hololens on for about 20 minutes at a time before they became uncomfortable and would push them up to the top of my head while I was not using them, like you do with sunglasses when you go indoors.

There are two key universal gestures that Hololens users will use all the time — the airtap and the bloom. The airtap selects and places a hologram (an object in the Hololens world) or selects a interface item like tapping a button.

Airtap gesture for selecting and placing objects in the Hololens

The key to this gesture is not to think of it as a pinch but as a tap with three fingers. Use your index finger to tap straight down and hit your thumb. Once you get the gesture right it works really well — it’s like getting good at typing on your mobile phone— those who can do it fast will rule the world :)

The bloom gesture is best thought of as the ESC or home button. Open your hand with it facing up. Like 🌺.

There’s an incredible amount to think through when working with the Hololens- from Unity development and C# scripts to world design and cognitive psychology of 3D interfaces.

It was so helpful talking to the developers who have already thought really deeply about the app experience on the Hololens. The conversations we had in conjunction with hacking away left me with two high level takeaways that I hope will help frame your development as you navigate this new frontier:

1) Be Tolkien

We are no longer building apps, with a set experience for users to walk through — we are building worlds.

While Tolkien spent dozens of years coming up new languages and territories for his worlds, you should start simple, brainstorming the rules of the world you want to create. Think about mixing real-world rules with a sense of delight and magic:

REAL WORLD — Make the experience intuitive by using real-world metaphors that users already know to guide them to the correct actions.

MAGIC — Surprise users by making your world magical, letting them go beyond the perceptual, cognitive, or physical limitations of the real world to something magical...

When learning about the galaxy, can a user tap on the object to learn more and then realize they can pick up the entire planet and throw it across the room?

When taking a tour through real life ancient ruins, can a tap magically make the scene of what they looked thousands of years ago come alive?

Plan for the rules of the world you create — and now, with mixed reality, rules for the real world of the user. More about this in number 2…

2) Develop with the logic of a space in mind

Get ready — this is awesome…

Spatial mapping provides a detailed representation of real-world surfaces in the environment around a users HoloLens. That means that you should develop story and app logic around surfaces that are seen in a users environment and plan to encourage your users to walk around their environment so it can be mapped.

Playing Fragments, a game from Asobo Studio for the Hololens

An app called Fragments is a first person detective game for the Hololens. It utilizes spacial mapping logic to enhance presence and story. Your window becomes a window in the story with a new environment mapped to it. If a couch or chair is found within your environment, then a character would sit in your real world couch or chair. If no couch or chair, they would stand. Clues in the game are often laid out right on your coffee table.

The Hololens can map real world surfaces with spacial mapping, but can also utilize logic around the user’s location and gaze in the space.

From helping the deaf hear, tabletop games, real-time physical therapy, finding and tagging triage victims, exercise games for astronauts, seeing real-time movement of people in a scene on your work desk, building for construction and IoT, our team’s social multiplayer charades game and so much more…the potential is incredible.

Holohacks San Francisco June 2016

It’s an incredible time for makers.

A big thanks to Microsoft, Dona, Pat, Kat, Ben, Steve, Alex, Rick, AR/VR Women, and my awesome Hololens Hack team — Sean, Xin, Mark, Anitha and Crystal #TeamHeadplay


Ann-Caryn is a UX designer + Co-Founder of BrightSky Labs. Previously a Producer with work on Discovery, HBO, Sundance Channel, BBC…etc., Professor of Cinema Arts and Media Designer at Sony Pictures.