What is Mixed Reality? And How Did I Learn to Iterate?

Last year, it was the year zero for virtual reality. 2016 then became the “V.R. 0001” since Oculus, HTC, and Sony are shipping their consumer VR headsets and controllers. And from my experience I think “A.R. 0001” still has a long way to go.

People usually think the difference of Augmented Reality and Virtual Reality in this way:

“I can see the real-time real world with virtual contents overlay on it, this is AR; I can’t see the real-time real world and everything is virtual, then this is VR.”

The reason why I mentioned “real-time” is because most of 360 videos shot in “real world” and look like “real world”, but it isn’t real-time. Augmented reality sees the real world; virtual reality can’t see the real world. Then “mixed” reality must be the combination of both. But how does Mixed Reality work?

Garden-Mixed Reality” was the Mixed Reality I co-created with my friends Vivek, KH, Nigel, and Wei. We started with an experimental project, called Project Gotan — trying to create an experience using Google’s Project Tango on the head-mounted display and make it immersive. And it ends up to become an amazing app we feel very satisfied, and now you can download on Google Play store for free. Some people describe this as an untethered long-range VR experience. Unlike Oculus, Vive, or PSVR, it’s absolutely wireless. It’s just And this is how I met Tony Parisi and became part of Wevr.

In case you didn’t know Project Tango is a mobile device tablet that has motion tracking, depth sensing and regular pixel cameras in front. The first priority for us was to make people feel safe and comfortable to walk around when people wear the HMD on their face. It was a huge challenge for us . It’s hard to make people safe when their eyes are covered by a huge display, and make them “want” to walk around?! No way! In the beginning phase of this project, people tended to just stand still and didn’t want to walk around. The most they would do was just rotate their head and body to see what’s around them, slowly and carefully. That’s it. Nothing more. However, when our system became more mature, people started walking for a small distance. But before they do that, it would take a few minutes to get used to the virtual environment. So this is what we do — since we are using Google’s Project Tango, a tablet that could scan physical objects, we designed a system that could scan the physical objects and make everything become virtual objects in the virtual scene, in real-time.

Real-time” is key for Project Gotan. In this way, people could actually know “what they see is what they see”. What they see in the headset is what they see in the real world, but becomes virtual. The system we created makes everything composed with voxels, or you can say blocks. Every block is around 10 x 10 x 10 cm (4 x 4 x 4 inches)in the real world. So if you see a bed with 80 x 60 x 8 inches like the one below, it would be a composition of 600 blocks with same shape of the bed but different textures on it.

Signature Sleep Contour 8-Inch Mattress, Queen from Amazon.com

However, after few more playtests(user tests), we think just the real-time scanning was enough to make people feel safe and comfortable to walk around in the virtual space. They still needed 2–5 minutes to get used to it. Most of them took off the headset before they started having fun with the environment. That’s not what we wanna see. So we decided to add something more — Pre-Scan system.

Pre-Scan is no more special than real-time scanning. Not at all. However, it helps us make people be more confident about our system. Why is that? It’s because we start with Pre-Scan, but, with Project Tango handheld. So player could actually see how our system creates blocks with real world, one by one. With the Pre-Scan, users could easily understand how it works, and tend not to hesitate to walk around when the headset is on their face. To find out this, we’ve been through tons of playtests. And this is what I learned the most in my school, Carnegie Mellon University’s Entertainment Technology Center(CMU, ETC).

Gotan Playtest with randon visitors

With all kinds of specialty in game development, ETC is a 2-year program grad school starts with a class called Building Virtual Worlds(BVW). In this class, we have to make a game or experience in almost every 2 weeks. So this is our schedule — make a game in 2 weeks(1), build a game/experience in 2 weeks(2), create a game/experience in a week(3), finish a game/experience with compelling story in 3 weeks(4), and produce a self-pitch game/experience in 3 weeks(5). We are always assigned to have different teammates until the last round. I have to say, this was a really exhausting for me, but, it was a great experience. And there were another thing happening in every round — you have to write down “You are good at…” and “You are bad at…” for all of your teammates, at least three of each. Oh man…that was another “great” experience. You would never know how your teammates describe you until you finish the last round:/

It could be emotional and mean…. I got a bunch of negative feedback, but I did learn a lesson that teamwork is such an important things to our life.

Schell Games — I Expect You To Die

Anyway, that was off topic. I just wanna tell you how important of that procedure for me to understand that. And that was how I learned Game Design. Not to mention, I also learned game design from a “famous” professor in our program — Jesse Schell, the CEO of Schell Games. Their VR game, I Expect You To Die, has been receiving all kinds of “Best VR Game” like Unity VR/AR Vision Summit and Proto Awards. He was the one got me involved to Virtual Reality. He is a humorous and brilliant professor, and a ridiculous speaker! You never feel bored in his talks. I bet you knew it after his talk in VRX in 2015— “40 Predictions of VR and AR in 2016, 2020, 2025”. If you didn’t, you should start from now.

Some people asked me about my opinion in mixed reality. It’s definitely the future, and a lot of people know that. However, technology is not there yet. The hardware is not strong enough to handle high resolution graphics and in the meanwhile capture the real world structure. It will take another few years to create a stronger/lighter tablet or a whole new HMD that is light enough to be accepted by consumers.

Back to the topic, what’s Mixed Reality? Microsoft Hololens disclaims that Hololens experience is Mixed Reality. However, from the point of “in AR you can see real world”, Hololens is an Augmented Reality experience. In my opinion, I think Mixed Reality is in between “you see the real world” but “you don’t see the real world”. Ah…What’s that?? It means using the real objects as a base and build the virtual world on it. But who cares? It’s just a term. Virtual Reality, Augmented Reality, or Mixed Reality, they are all blue-sky. Maybe Virtual Reality is less blue-sky now, but still a lot to explore. No one would know what will happen in 5 years. Who knows what’s the best for a Virtual Reality experience? Who knows how to make a greatest Augmented Reality game? But all we can do is, keep trying, until people like it, and buy it.

*Download “Garden — Mixed Reality” from here if you are interested in.