The State of Reality: Virtual and Augmented
In light of steadily increasing amounts of news around virtual, augmented and mixed reality, let’s take a look at the current state of things and slowly make our way into the future of the two technologies in this this three-part miniseries of articles.
To start it all off, some clear distinctions between VR and AR, in simple terms.
Virtual Reality offers its users an entirely virtual environment to explore and interact with., everything you see has been created from scratch by a team of designers, artists and programmers. Think video games or the concept behind the movie Ready Player One. When talking about VR we think of it as being a set of hardware components, with a headset that isolates its user from the outside and some controllers in the shape of joysticks.
Augmented Reality experiences overlay virtual information over real-world objects using spatial registration -i.e. it picks up your real world environment- and then anchor¹ said virtual assets to physical ones while allowing users to interact with them. Even though you can find headsets ready to purchase which are sometimes referred to as ‘AR headsets’, when discussing about AR technology we’re thinking more towards the software side of things. Headsets along with the AR software that runs on them provide a mixed reality experience, or MR, but at the moment the two terms are used interchangeably.
Vas a technology has had a bumpy ride navigating the infamous emerging technology hype cycle ever since the beginning, when HTC came up with the Vive followed by Oculus’ response with it’s own Rift system shortly after back in 2016. In the following two years the technology advanced only so much from these v1.0 devices, content for these platforms remained scarce and users were slow to adopt them. Until now…
Enabled by incremental technological advances in individual components, thanks to heavy R&D investments by multiple companies, and as consumers’ tastes become increasingly more clear, headset manufacturers have taken one of two paths in designing their v1.5² hardware: 1)Pushing the envelope in terms of quality with top-spec devices or 2)Making a simpler, more accessible product.
The first approach -embodied by devices like the Valve Index or the HTC Vive Pro- is taking a bet on future demand coming from Pro-level users or the industrial and enterprise spaces, as it requires significant financial investments from its potential buyers. It’s easy to dismiss this approach as it only covers top-end users with proportionally top specs and prices to match, but pushing the envelope in terms of quality is something the industry definitely needs at this stage in order to evolve at a steep rate in the coming years. These systems are also the ones that usually improve the VR experience in secondary directions, away from the base requirements, -e.g. pressure-sensing controls, improved space tracking etc.- features that will migrate into the low-end category of hardware once they will become cheaper and smaller.
The second plan of attack, addressing an exponentially larger audience at the low-end of the consumer scale with a good-enough product, looks the more promising of the two paths in terms of adoption potential if we judge from the reaction to critically-acclaimed devices like the Oculus Quest and Oculus GO. These devices have lower costs -$399 and $249 respectively-, are trivial to set up and, perhaps even more importantly, start up, have no need for a separate computer to run the heavy computational aspects that come with VR and thus eliminate the need to be tethered to a stationary PC via hefty wires. Each of these changes individually⁴ would have represented but an incremental change to an old device but, put together, they make for a completely new experience and are bound to spur a new species of VR experiences, one that is mobile-first; Mobile VR⁶, that is.
A hardware devices on the other hand, are few and far between in terms of availability for the average Joe right now. Devices dedicated to be used primarily as outlets for AR originally became available to the public -albeit in a limited release- with Google’s Glass headset in 2013, followed by a, more rounded in terms of features, Microsoft HoloLens three years later. After Google decided to pull its product from the shelves and focus more on industrial and factory-type applications, Microsoft soon followed suit and the sector went silent until Magic Leap launched the development kit for their Magic Leap One system in 2018. At the beginning of 2019 both Windows and Google announced the release of their second generation devices, again aimed to be used by ‘first-line workers’, people who aren’t traditionally working at a desk, but instead do some form of manual work that involves high complexity -e.g. assembling a device with complex components, a doctor operating on someone etc.
There are multiple reasons why both these large companies retreated from the consumer market into the factory or other, more controlled, environments and they can be all be summed up with this: the technology isn’t ready yet. Mixed Reality headsets are heavy, their battery life is not great, computational power is limited and there’s little content or practical application for the technology. The bottom line is, the hardware has a long way to go before its usefulness and convenience become apparent to mainstream users and, as a result, before software developers can or are willing to invest their efforts into the platform.
Regarding future developments on the hardware front, I see these technologies heading in several different directions, also depending on how far away in the future we want to look.
On the VR front, I see devices continuing to evolve along the same two paths in the short term -i.e. high-end vs. low friction. As time passes, hardware from top-tier devices will continue to be miniaturized and optimized, slowly trickling down to the easy to use mobile VR devices while, at the same time, some of the translatable ease of use elements will migrate in the opposite direction. As soon as the VR experience on these low-end devices becomes acceptable, manufacturers, followed by developers, will gradually move away from the expensive hardware and prioritize their designs for convenience and accessibility. In this respect, a company like Oculus is perfectly positioned to detect when exactly this shift in technology and consumer preference will actually happen, as they sell and manufacture both solutions simultaneously. When the sales trend shifts away from the tethered Oculus Rift in favor of the Quest and/or Go devices, the company will waste no time optimizing it’s production output for the better-selling products in a move towards conquering the vastly larger Mobile VR -or mVR- market.
Another almost inevitable development I see happening in the foreseeable future is a temporary merge between technologies inside mobile VR and mixed reality headsets. As decent display hardware for MR devices is slow to develop, VR incumbents will start producing hybrid VR-MR devices that feed the real-world environment into your VR headset via stereo cameras placed on the front of the headset. I see this as a a natural intermediary step that will enable an experience with improved graphics on the digital assets than existing MR headsets can provide while the obvious sacrifices in terms of weight, subpar controls and diluted feeling will most likely be overlooked.
Multiple up and coming technologies being developed right now will help AR and VR along their way of being adopted by the masses and the most important ones will prove to be those that enable users to have more freedom in their experience and those that reduce friction and add convenience.
In order to achieve a seamless blend of virtual and physical objects, mixed reality devices need a giant leap in terms of understanding the real world, a leap that can only be provided by Artificial Intelligence powered by machine learning. Right now, sensors can only pick up the physical size and shape of the objects in your living room but don’t have the ability to ‘understand’ whether a particular volume is a coffee table or a couch. This severely limits the perceived realistic experiences available to be developed in mixed reality, as the human brain expects the interaction to be different based on the nature of the objects interacting and delivering anything else breaks the immersion feeling.
This ability for semantic understanding will also solve another issue mixed reality headsets currently have when it comes to manipulating virtual objects in a real world context. Using physical joysticks was never a realistic option outside of virtual reality, as non-intuitive controls remove the user from the experience and break the illusion of having digital objects in your physical space. This can be easily solved once mixed reality headsets can accurately track and understand the exact positions of your fingers, with the optional addition of being able to interpret voice commands received via natural language. Voice commands are becoming more and more reliable as voice assistants are used and AI algorithms are trained with increasingly more training data. Apple introduced its ‘Voice Control’ feature into it’s next version of macOS, further pushing competitors to follow suit.
Another technology that has the potential of taking away the need to have massive computing power on the wireless headsets themselves -thus reducing weight and heat production- is streaming. Google’s Stadia streaming service will allow gamers to run the computational side of games on a remote server and stream it to their computers at home, thus reducing the user’s need for renewing their hardware every time a game comes out. The service launches in November and, if successful, scaling it to work with existing VR and MR headsets will only depend on your internet bandwidth. This innovation could have other potential effects on VR like more Pro-level headsets going wireless and prices going down overall while MR hardware might become simplified, with less bulkiness on the head-mounted display and perhaps a complete lack of a separate tethered unit in charge of computing.
One more technology with potential great impact on Mixed Reality headsets in particular is making waves right now is 5G. If we want to walk around with these things outside our homes while seeing fast, ultra-high quality experiences, 5G is a technology we definitely need for this task. While today's almost ubiquitous 4G allows us to stream video in HD, devices of the future will need the capacity to move twice⁵ the amount of information plus, probably, simultaneously sending inputs from the user to the cloud in the shortest time possible. Since the speed of implementation of 5G will probably take between 8 and 10 years and unlikely to be deployed outside of metropolitan areas due to the costs involved with the new infrastructure needed to be laid down, we would also need a backup system for those cases when a cellular network is not accessible at all. This backup will probably be provided in the future by multi-satellite constellations offering internet access with relatively low latency like SpaceX’s StarLink.
If you enjoyed this article give us a follow here on Medium and on Twitter
¹ Some sources out there insist that anchoring digital information to a physical one is actually Mixed Reality but I personally disagree with this differentiation. In my views MR is augmented reality delivered via dedicated hardware.
² I prefer to call the recently announced devices a v1.5 as opposed to a full v2.0 because they entail nothing revolutionary in terms of features that would potentially change the experience in a significant way.
³ While existing VR headsets use other features, like FieldOfView or different Control Joysticks, to market their individual systems, there is little visible differentiation on those aspects between brands and said differences are hard to notice if the individual experience is deeply immersive. The type of panel used for the headset can sometimes improve the experience but the shortcomings of inferior panels can be overcome with software, smarter sub-pixel arrangement or experience design.
⁴ Windows tried to reduce the number of external sensors needed for a VR headset with its Windows Mixed Reality standard but instead created more problems than they solved. The prices of these headsets very soon became non-competitive after V1.5 devices came out and users got confused by the deceiving terminology as all the devices actually offered VR experiences and not MR.
⁵VR and MR headsets have to project images in each eye independently in order to create the stereoscopic effect, for this they need double the amount of resolution compared to a TV.
⁶Some prefer to refer to this as ‘Standalone VR’ in an effort to differentiate from headsets that used a physical smartphone as a screen but in the grand scheme of things I think those will be but a smudge in the VR history books, one to be forgotten.
