The Magic of Mixed Reality

Magicians are in the business of making things appear out of thin air: handkerchiefs, rabbits and sometimes attractive assistants.

Augmented Reality (AR) also makes surprising things appear, but by way of technology. At first it can seem like magic too: a Pokémon suddenly appearing out of nowhere in the street-corner.

Pokémon Go with AR turned on (left) and off (right)

Pokémon Go uses AR to showcase the basic premise of the game: that our physical world is actually populated with Pokémons. But as avid Pokémon Go players know, pretty soon you turn this feature off. You got the point, thank you very much, and you want to play quicker without AR. This kind of “light AR” is a neat trick but still quite limited.

“Mixed Reality” is a more powerful experience. In Mixed Reality the virtual objects or characters don’t just appear and hang mid-air. They are three dimensional holograms that connect with their physical surroundings. It’s a stunning experience and it doesn’t wear off.

A virtual character sitting on a real sofa (from Fragments by Asobo Studios)

In Mixed Reality holograms are programmed to connect to existing surfaces or furniture. A virtual dog can roam around the floors of your office or a virtual human character can sit down at your kitchen table or prepare a virtual dish at your stove. Mixed Reality experiences combine the physical surroundings (the floor, the kitchen, the stove) with fully-formed virtual objects and characters (holograms).

Spatial Mapping

Mixed Reality is possible because the device builds an accurate model of the user’s surroundings. The system knows exactly where the floor, walls and ceiling are located and where tables and chairs are standing. Devices like the Microsoft HoloLens distinguish between separate types of furniture (a chair is different from a sofa) and e.g whether they have a hard or a soft surface. Spatial mapping is now getting very good and very fast as machine learning is kicking in. The more spaces are scanned, the better each following scan is going to be.

A mesh model of the space created by HoloLens.

The Unexpected

Mixed Reality connects you to the unexpected. Think about it: virtually anything can be brought into your physical space. Then combine this with spatial audio. You can make a small virtual character to walk on your work desk and then parachute from the table down to the floor. Or you could invite a talented musician to sit on your living room couch (as a hologram) to give you and your friends an intimate unplugged concert. The range of experiences that can be done in Mixed Reality is limitless.

A scene from HoloLens game Fragments.

Natural Experiences

By contrast, in Virtual Reality the experience takes place in an entirely virtual space. This is why immersion in VR sometimes feels almost too complete: you’re immersed but your persona almost dissolves into the experience. You might feel as if you’re “losing yourself”. This can be interesting in itself and is often the whole point of a VR experience: to explore a new identity or a new kind of reality.

Full immersion is fascinating but mentally taxing. I struggle to stay “fully immersed” for longer than ten minutes even in the best VR experiences.

Playing Minecraft on HoloLens.

My experience of Mixed Reality is very different. You don’t “lose yourself” in Mixed Reality. I continue to see my own body and I feel at ease and I can walk around freely.

Designing a 3D object with HoloLens.

When holograms are skilfully connected to your own world, the effect is stunning. And you find yourself engaging on an immediate and emotional level.

That’s the real magic of Mixed Reality. It doesn’t feel like magic. It feels natural.