The future of Mixed Reality and probably how we are going to use it

Ideation #10

It’s been a while since I last wrote anything. I guess it’s contextual right? Before I jump right in, whatever written here is totally my own individual views, not affiliated with any companies.


Mixed Reality. Is a new tech I’m betting on. When this happens, we won’t be needing a smart watch. Cause it’s going to be an outdated device, in my opinion. And people are going backward, wearing old expensive timepiece. This time with digital overlay with their new smart… lenses? Something that overlays digital with reality? Okay, let’s call it… glasses. Smart Glasses.

Before we go further into the future, let us recall roughly about 5 years ago in the tech industry. The period when most tech visionaries and technologists thought Augmented Reality is going to be a big thing. But the reality of lifting up your device with camera feature switched on on the screen and pointing towards something to get information didn’t grow in us. Why? I don’t know. Human nature.


Right now, with Facebook acquired Oculus Rift, companies working on their VRs like Google, Razer, Valve, HTC, and many other companies working on VR accessories. Virtual Reality is even crazier than AR years ago. From my point of view, VR is an immersive experience product, a peripheral for gamers, tech enthusiast and probably monitor replacement for some. But I guess is not something for everybody that would use on a daily basis?

A friend of mine showed me this gif and he was spot on. I think my wife will give the same expression as the lady on the right. It kind of look stupid from another person’s point of view.

My gut feeling tells me that, on a general consumer level, VR’s fate might end up like Xbox Kinect or PlayStation Moves (PS3). If VR releases a huge number of contents on day one, consumers not picking the hardware and games fast enough, developers will not be able to see returns from their first project and the loop goes on in a declining state till they stop working on it. So it’s going to be just two options, either it go super crazy like it did with mobile phones OR it just going to stop with tech enthusiast.

Hang on, I’m not trying to down-sell VR. What I’m trying to say is, it takes a culture for everyone to adapt and everything will follow. Just like QR code, it’s everywhere in China but not really the case with the rest of the world. VR is great, I mean really really great. But it has to be used at the right place and correctly. Some good use of VR right now are probably simulators and theme parks. For example, if Universal Studios Singapore were to replace the 3D glasses and screen with VR headset in Transformer the Ride. It’s going to be a blast.

Bandai Namco’s The Idolmaster: Cinderella Girls Viewing Revolution https://twitter.com/hasyaP/status/786455959908388864/video/1

How about watching a live concert at home in VR by purchasing virtual tickets? And… maybe they are in 3D anime characters? I’m not judging, you know, as long they enjoyed themselves.

Mark Zuckerberg live VR demo at Oculus Connect 3 https://www.youtube.com/watch?v=NLnk2cCirQM

Recently at Oculus Connect 3, Mark Zuckerberg and Co really nailed it during the demo. I can’t really explain why but probably they humanized it. The cartoonish avatar that represents your actual self in the VR world just blew my mind.

WIRED: The Untold Story of Magic Leap, the World’s Most Secretive Startup https://www.youtube.com/watch?v=BLkFWq_ipCc

Meanwhile, when everyone is rushing into the VR goldmine since oculus rift made popular on Kickstarter. Microsoft was making the holo lens which is an augmented reality glasses. As well as Magic Leap making their own similar device. Well, both companies are trying to stay away from the word augmented reality which sounds dated in the tech industry. So they came up with the word “Mixed Reality”, where you mix both virtual and reality together. No pun intended. What I believe is that augmented reality was the period where footage from camera merging with computer graphics. Whereas Mixed Reality is referring to computer graphics being put on top of what you are actually seeing.

HYPER-REALITY by Keiichi Matsuda https://vimeo.com/166807261

This mixed reality video concept from Keiichi is close to Microsoft and Magic Leap are doing, but it seems a little too overwhelming that I almost puke at the future. But that’s not what I want to discuss. What I really want to point out, are these trackers/QR codes. I don’t imagine that we will be replacing traditional print graphics for the sake mixed reality information. Instead, we should be embracing our existing static printed billboards, posters, signages and packaging, using them as a new age matrix barcode / tracker.

Movie poster of Marvel’s Doctor Strange loads movie trailer from Youtube and ratings from IMDb and Rotten Tomatoes
  • Imagine looking at movie poster, the video trailer instantaneously loads up floating above it without the need of a physical TV screen and speaker. It can be contextual to whoever is watching, say a Korean looking at a Hollywood film trailer. The narration should be playing the Korean dub trailer.
  • While looking at the food packaging, you can quickly gather information based on your personal settings. Say, ingredients that you’re allergy to it, promotions, price comparison and more.

I believed by the time Mixed Reality gets in the hands of consumers. Image tracking would be improved tremendously.


Apps

Now, think about what we could do with it using today’s mobile app or specifically for this concept? The Smart Glasses just need two key trigger points. What to recognize and what to show?

Mock-up of how we can easily receive information just by looking
  • Bus Arrival Time 
    In Singapore, Land Transport Authority, our local government agency released an open API of bus arrival timings for developers and accompanied by a consistent bus signage across the country. By looking at the bus signage, it displays the list of arrival timing without pulling out your phone and launching the app.
AR translation from Google Translate app https://translate.google.com/about/intl/en_ALL/index.html
  • Translation
    Traveling overseas? Looking at foreign languages you don’t understand and replaced it with your primary language on the spot. Just think Word Lens (now part of Google Translate), but without the need to point your phone to what you want to translate. Just look at it from your Glasses and the app will replace them.
Psyduck in Pokemon GO
  • Pokemon GO
    How about being able to spot a Pokemon through your Glasses?
An example of Waze in Mixed Reality
  • Driving Navigation HUD
    What about getting into your car, it removed info from other apps and receive all your driving information needs through the glasses? Turn by turn navigation, map, speedometer and traffic information.
Mockup of how we can spot our Uber ride
  • Recognizing your Uber ride
    Not driving? Well most of us, the Uber rider will need to look through the info given in the app to look out for our ride. What if the Glasses can pinpoint the car you’re looking for? Based on the car plate detail, brand and model. And quickly point out “Hey! Here is your ride!”.
The crosshair appears when you’re near your car and follows where you’re looking. By aligning the crosshair to the MR lock, you can trigger to unlock by double tapping your fingers without your hand in the line of sight of your glasses. (Will come to that interaction below)
  • Car Key
    Unlocking your car by aligning the MR lock with your eyes?
  • Mobile device second screen
  • Desktop app switcher just by looking at it
  • Characters emerging from your trading cards as you play with them
  • Kids playing with physical toys, simply just add extra visual effects on top of it
  • And the list goes on and on…

Think of it as a hands-free accessory where little action is required to receive information and trigger an action. What if we go beyond visual contents by adding more components to this device?

1. Audio Feedback without blocking out the world
Bone Conduction Audio — is the conduction of sound to the inner ear through the bones of the skull.

Using this technology allows the individual to hear sound without plugging themselves away with earphones or headphones all day just to listen from the device. So whatever you’re hearing from Mixed Reality is just for yourself, from watching movie trailers, voice of your digital assistant and other audio effects, all and still have the freedom to hear from the reality.

Universal Studios Singapore — Shrek 4-D Adventure

2. Physical control and feedback, the illusion of touch
I remember this experience I had when I was in Universal Studios Singapore, watching Shrek 4D. That very moment when Donkey sneezes straight at the screen, water was sprayed on our face, the entire crowd including me goes “eeewwww”!! That extra dimension was a nice Touch. Pun intended!

Apple iPhone 7

Air gesture never seems right to me since Microsoft Kinect launch, simply because there wasn’t any physical feedback that our brain needed as to physical button, keys and touch screen. Today, there are quite a number of devices that are using haptic feedback to mimic certain physical interaction. For example, Steam Controllers touchpad mimics a scroll wheel as you slide it down using the haptic feedback. Apple iPhone 7 home button mimics a button press, even it is flat, as well as all the iOS dialer, scroller and other digital components.

So are we able to interact the Mixed Reality world and have physical feedback? Imagine you have a wristband and this device comes with Google’s Project Soli technology and haptic feedback built in. Now you could just look at the floating MR UI and interact with it. Select it, scroll it, slide it, all the simple familiar gesture that we are used to and on top of that, we get to feel that imaginary button or knob. Hopefully, that haptic feedback vibration could reach the fingertips from the wrist.

Google Soli

How about expanding the UI beyond your watch small screen using Mixed Reality?

Tom Clancy’s The Division — E3 2013 gameplay reveal https://www.youtube.com/watch?v=njfj6KwEAfg

How it should look like

I’m not trying to complain here but what Microsoft currently have, Hololens, still looking weird like a VR headset from a third-person point of view.

But what I’m envisioning it would be similar to the device used in Westworld. Surprisingly, it was almost identical to what I imagine it would be. Notice the additional frame extended out close to her ear? I’m guessing bone conduction audio?

Elsie Hughes in Westworld Episode 2

Mixed Reality aside, one feature I would like to have, especially from someone who wears glasses all day, is the ability to change from clear lens to shades on demand. I knew transition lens exist, but I had one problem that bothers me every time I’m in a photo under daylight. You won’t be able to see my eyes.


Privacy and Security

The downside to this it’s also privacy. Should we allow capturing video footage through the glasses like Snapchat Spectacles? It comes with a light indicator that, when the wearer is recording or taking a picture, it turns on just like a webcam. Or maybe it shouldn’t capture anything. It’s hard to justify this as there are hundreds of different use cases where some scenarios are handy and some are just plain “glasshole”.

Lastly, the most important part of technology, security. How do we introduce security into such device when it knows almost everything about you and yet not being disruptive on your daily usage? If it’s possible to track where you looking, is it possible to scan your iris to verify you’re the owner of the smart glasses you’re wearing? Gawd imagine you have to key four-digit password haha.

What would the future be? Would the dust settle down on VR like it did to Microsoft Kinect? Would MR pick up the pace? Maybe both will deeply embed into our lives? Or a possible hybrid between MR and VR? From my point of view, MR will be used on daily basis and VR is an entertainment device. Both futures are unpredictable. People only knew what they want until you put it in their hands.


Update 18 May 2017

Google did it! on the phone…


EndNote

You have come this far and I thank you for taking the time to read this article on VR and MR. If you have any question in mind, do feel free to drop me message here or Twitter @theglimy