AR Glasses, Step Back: VR/AR Hybrids Will Lead Mainstream AR
Passthrough AR technology solves many of the current-day issues of AR and will pave the way to mainstream AR in 2021
Over the past months, you may have seen GIFs like this circulating around the web.
Cool, right? These are AR experiences (we’ll describe them as AR on this article for simplicity’s sake) designed by developers from all around the world — but they weren’t built with traditional AR headsets like the HoloLens, Meta 2 or Magic Leap. They were built with traditional VR headsets like the Vive Pro.
These headsets are special and contain something called passthrough AR — this is an experimental, dev-only mode that lets creators access live footage from the front cameras of the device and then merge in digital elements on top of the footage.The result is a convincing illusion that makes you forget you’re wearing a headset, mixing your physical world with the digital into one: a Mixed Reality experience.
Passthrough AR isn’t perfect right now: the positioning of the cameras needs some work to simulate your vision and the latency isn’t low enough for day-to-day use. But devices running it right now were never designed for passthrough — it has always been treated as an afterthought, a bonus, experimental feature.
But this won’t be a problem for much longer.
All major companies in the space have been dialing up their experiments with passthrough AR — and some even started building their own headsets. One of the most notable cases is VRVana, a startup based out of Montreal who got acquired by Apple for their VR/AR hybrid — and could be a signal of how Apple is considering entering the immersive game.
VR/AR manufacturers are finally starting to realize the power of passthrough AR, and the list of benefits isn’t small: it enables full-fledged AR experiences for a fraction of the cost, addresses many of the issues with contemporary AR hardware, collapses all VR/AR/MR content into a single library and gives developers more creative power than ever before.
This is the beginning of a new era: VR/AR will finally become a single, immersive medium, setting off a new wave of creation that will start pushing AR into the mainstream.
And it will all begin with hybrid VR/AR headsets.
Why AR glasses are a mistake for the consumer market (right now)
Let’s look at the main issues AR headsets face today:
- Low FoV (30°-40°) due to the physical limitations around modern see-through lenses.
- Limited space to place all necessary processing power & sensors while retaining a lightweight form factor that doesn’t cover your face
- They don’t look “cool” by mainstream standards
- Requires a ton of custom technology that needs to be internally developed (which is why they cost thousands of dollars)
The thing about most of these issues is that they don’t necessary come from building a AR device — but from building a AR device that holds some resemblance to glasses.
All main AR companies today are striving for the same final goal: they want to create AR glassware, computers so lightweight, so transformative and so unobtrusive that you’ll be using them all the time. It would be the biggest consumer electronics revolution since the advent of the smartphone, essentially replacing all of the screens and electronic devices we use on a day to day basis.
It is an admirable goal, and I am a big believer that we’ll be getting there within the next decade. But when your glasses are your final goal, it changes how you build things in the present — and that can get you cornered in unexpected ways.
For one, consumers know what glasses feel like (both from the perspective using it a well as from the perspective of looking at someone using it) — so those are some pretty high expectations to go up against. Your device needs to be super small, lightweight, allow for normal social interaction, feature see-through optics and somehow hide all of the sensors that make the device function while making it look palatable.
AR headsets don’t succeed at most of the above criteria. While their see-through lenses work, they’re packed with technical limitations while also darkening your surroundings and blocking your eyes from view (making social interactions weird and robotic). The form factor is an engineering marvel, but it’s in a weird HMD uncanny valley territory of “Not quite a set of glasses, not quite a headset.” that makes them feel odd to consumers. Goggles? They’re not sure and I’m not either.
So if we don’t have the technology to create glasses for the consumer market, why try to do them at all?
And here lies an opportunity.
How VR hybrids offer a solution
So what if we briefly abandoned the “glasses approach” and just tried to create a AR device that allowed consumers to play AR experiences at an affordable price. What would that look like?
It would be a AR-passthrough device — a VR/AR hybrid. Here are the reasons why:
- No FoV issues. AR-passthrough devices use traditional displays, allowing users to enjoy wide-FoV experiences.
- Black Is Back: The color black cannot be rendered in AR headsets today. Really. This fixes it.
- A More Flexible Form Factor. VR headsets work with a different set of expectations when it comes to distribution of weight, sensors, form and look. This gives engineers a lot more freedom to work with.
- Cheap — Most of the technology used in a AR-passthrough device would be off the shelf hardware recycled from the smartphone era. This makes AR-passthrough devices much more affordable than their AR counterparts, which require a ton of custom tech. This would be the easiest way to achieve a fully standalone MR device for a ~$500.
- Unification of VR & AR libraries —AR doesn’t need to start from scratch as a medium. This unification of VR+AR lowers market fragmentation and allows users to buy a single headset with an extensive and already proven VR library — as well as all the AR experiences that become available.
- Styling Options — This is true specially of most recent headsets like the Oculus Go and the Oculus Quest — since the advent of inside out tracking, the front of the headset is no longer covered by sensors how it used to — the front of headsets are now empty space begging to be repurposed. This opens up a ton of creative and engineering opportunities to make VR headsets that look stylish and incorporate technology in new, unique ways.
But there’s another, really important sixth reason that is only obvious if you’ve been creating VR/AR experiences…
If all the hardware and sensors are equal, you can make better AR experiences in passthrough AR than see-through AR
To understand that last point, let’s compare the creation of VR-passthrough content vs traditional AR content.
See-through devices like the HoloLens have additive displays. This means that as a AR creator you design a digital layer that gets laced on top of what the user sees in the real world. This layer has a black background that gets cut out, and everything with color gets projected into space.
This is cool, but has some limitations. What if I’m making a noir thriller and I want the user’s entire world to be black and white so I can convey a greater sense of mood?
You can’t do that with additive displays. There’s no way you can make someone’s entire vision B&W. But you can with pass-through VR.
When designing passthrough AR experiences, you’re not just designing a layer to be placed on top of the world. You’re merge your content with the camera input and have final say over how everything looks — you have control over every pixel.
So if you want to make the whole scene black and white, you can integrate your digital content with the camera feed and then apply a black and filter on the top of the whole thing.
This opens the door to much more nuanced AR experiences in which creators have a stronger control over the mood, look and feel of their experiences — and better pure-VR experiences as well.
This also gets rid of some limitations around additive displays — if you’re designing an experience for the HoloLens or the Magic Leap, for example, you cannot use the color black since it just becomes fully transparent.
But even with all the benefits, passthrough AR isn’t perfect yet. Because you’re looking at the world through the lens of a camera, image quality and the resolution of HMD’s start to matter. The real world won’t look as good through a camera as it looks through natural vision, but I think the creative freedom this gives creators over the look of the world compensates for that limitation.
What this means for the future of AR
So the question is, what is going to develop faster: AR experiences powered by AR-passthrough or AR experiences powered by additive see-through lenses?
My guess is that in the next 5 years, VR/AR hybrids powered by passthrough streaming will be king among customers and the beginning of affordable, mainstream AR. This approach circumvents a number of issues that hold the medium back, provides users with a better experience, gives creators more creative freedom and consumers a much larger bang for their buck. Creative, UX and economic realities all conspire for its success.
But this won’t last forever. The years that follow will start to finally bring the rise of the glassware that was promised as the surrounding technologies mature. Ultimately, the whole industry’s north star is the same — the question comes from what’s the best way of getting there.
Different companies will bring with them different approaches. VR manufacturers will embrace the passthrough movement with their headsets and start pushing it into their large consumer base (Google, Oculus, HTC and Apple all have been experimenting internally with it). Other companies like Magic Leap will approach the consumer market from a totally different angle — and some companies like Microsoft will attempt to do both simultaneously .
All of them have a role to play. Regardless of whether we’re ready for it, we need companies trying to make AR glasses to push the industry forward.
But there are other ways to bring AR to market, and an AR passthrough wave is on the horizon. AR will come to the masses — but perhaps not in the way we expected.