Analyzing VR as a computing plaform

VR clearly has a big future in games and interactive entertainment. For many folks, that’s plenty — they’ve been waiting to jack in for years. But there’s a larger story waiting in the wings, which is that VR has the potential to become a genuinely new computing platform, one that will reconfigure our world as deeply as mobile computing.

While this broader promise has been obvious for decades, and has figured prominently in fictional accounts of VR, the actual path to widespread adoption and market impact is a lot murkier. What will be the killer app? Which use cases will break into the mainstream earlier, and which will lag behind?

In this post, I’ll take a look at the necessary knowledge and capabilities that underlie various consumer applications of VR. Here is a diagram that summarizes the analysis, which we’ll unpack below:

Each VR application depends on some combination of underlying technologies and design know-how. We can analyze these dependencies to make educated guesses about when it will be possible to bring products within each application to market.

Technological Underpinnings

VR is not one technology, but many, and these technologies are at very different stages of development and maturity. The various consumer applications of VR draw on these areas in quite different ways. So any analysis of their time to market must first account for these technical dependencies.

I contend that there are four broad areas of technology that are directly relevant to VR as a general computing platform.

• Intent capture is the ability to reliably and fluidly understand what the user wants to accomplish in any given moment — input, in other words. Until very recently, consumer VR was sorely lacking in this area; recent advances at Valve (Lighthouse) and Oculus (Touch) should improve the situation greatly.
• Persona capture involves sensing, encoding, and representing the relevant aspects of the user’s appearance, behavior, mood, nonverbal cues, and so on — whatever signals are needed for the application at hand. This is the least mature of the technical domains, especially when it comes to solutions accessible to the consumer.
• Environment capture refers to the ability to sense, encode, and represent real-world environments. These technologies are maturing rapidly, with intense R&D underway at companies like Jaunt VR, NextVR, GoPro, Matterport, and more. These capabilities may become commodity in short order.
• Environment rendering is what most people think of when they think of VR: the ability to render virtual worlds with high quality, including all of the relevant sensory modalities. Modern VR arrived when Oculus demonstrated that excellent rendering would be available in consumer-ready devices.

Of course, each of these areas itself comprises many individual technologies, and any true assessment of the near-term potential of a particular VR application will need to drill down into more detail than we can cover here. As just one example, environment capture can be divided between (relatively) simple stereoscopic 360-degree video, and full scene geometry capture — the latter requiring the application of much more sophisticated computer vision techniques.

Design Factors

But it’s not just the raw technical capabilities that determine whether a given use for VR is ready for prime time; it’s also the collective knowledge of the product development community about how to use these tools to the greatest possible effect. This is the design question, broadly conceived, and it can be broken into the following three areas:

• Usability covers all areas of UI/UX — how can users make their intentions known easily and efficiently? This is the companion problem to the technical challenges of intent capture.
• Narrative concerns our ability to tell stories, fictional or otherwise, that are informative, engrossing, and that invoke appropriate emotions.
• Information presentation refers to our expertise in depicting ideas, data, schematics, and other abstractions; think charts, infographics, technical drawings, and so on.

With, say, capacitive multitouch, it took several years for the design and interface patterns to get sorted out. While eager developers will push through this phase as fast as possible, it will take some time. Some of the design principles from other media will hold up, but others will need to be adapted or rethought entirely. Lots of bright minds are already immersed in these problems, with more waking up to the fun challenges all the time.

But make no mistake: all of the design domains — usability, narrative, and info presentation — are in their infancy when it comes to VR.

Applications

Now that we have the technology and design enablers mapped out, it’s time to turn to the applications. I have a dozen shown, but there are sure to be more. For each one, we can characterize the strength of its dependence on these areas — indicated by the thickness of the lines in the diagram.

Some applications are primarily gated by technology, and they really won’t be viable in the market until the basic capabilities are improved. In these areas, new ventures will need to take on that basic R&D if they want to get started now. Personal communication is a good example here: we just don’t yet have consumer-ready solutions to the live persona capture problem. Once these capabilities are developed, however, this space will be cracked wide open.

Other areas are more constrained by our design knowledge; we don’t know how to create experiences that really deliver the goods. I think productivity apps are a great example here. Now that Oculus and Vive are both on the verge of bringing excellent hand-tracking input controllers to market, the primary remaining technical blocker — intent capture — will soon be removed. But what are the VR equivalents of Word, Excel, Powerpoint, and Photoshop, and how will they work? I would argue that our collective cluelessness in usability and interface design is the main gating factor.

Given all that, which applications are ready, or nearly so, and which will take some more time to bake? Here is my current stab at the likely timelines on which each of these applications will be able to be brought to market.

Short Term (9–18 months)

• Games: The strongest dependency, of course, is on rendering, which is quite mature. Other dependencies are weaker, and can mostly be worked around in early generations. Narrative will suffer in the short term as the community learns how to establish and develop story lines.
• Live events: Aside from rendering, the main dependency is on environment capture — but this is moving forward really fast! Other factors, out of scope of this article, will be crucial: content deals, distribution platforms, and so on.
• Real Estate: This application is almost entirely gated by the ability to quickly, cheaply, and accurately capture interior environments. Matterport and others are on the case.
• Journalism: On the one hand, effective VR journalism will require a conjunction of environment capture and narrative technique — both of which need further development. But early efforts can probably succeed even with the capabilities already on hand, and gradually grow to include the latest developments.

On the horizon (1.5–3 years)

• Productivity: These apps live and die by their ability to elicit, capture, and respond to our intentions. Accurate and efficient input solutions are just now becoming available, and it will take some time for designers and developers to figure out how to use these tools to good effect. While this application is listed as medium term, many of the most important entries will take years to emerge and refine.
• Communities: Getting together with other folks to talk about common interests — it’s one of the first and most enduring uses of the internet. To do this right in VR requires both decent persona capture (the main blocker) and product design, but this application should be addressable in relatively short order.
• Cinema: Story-telling does not face major technological blockers, now that full-field cameras are rapidly becoming available. The main gating factor at this point is creative — how to tell stories in this medium. Thus the intense efforts by Oculus, Google, and many traditional studios and production houses to explore and educate.
• Education: A bit hard to analyze, since different subjects will have quite different requirements. History and literature will share many qualities with cinema and travel, while cell biology and skills training might feel much more game-like.
• Travel: This is almost a pure environment capture play — it’s all about making us feel like we’re there, with as much fidelity and leveraging as many sensory channels as possible.

Further out (3–5 years)

• Business communication: The requirements here are complex, spanning high-fidelity interpersonal relations, presentation of information and ideas, and solid product design. While current conferencing solutions like GoToMeeting are terrible and impose great pain, there’s a lot that needs to come together before individuals and companies will switch.
• Personal communication: This application is far and away the most sensitive to the nuances of persona capture, including all of the nonverbal communication channels that people use. Satisfying solutions will not emerge to displace existing communications methods until major progress is made here.
• Expanded perception: Another application that’s tough to analyze — granting us new perspectives on the real and virtual worlds through novel presentations of data and sensory input. It will take some time for the design principles to mature, and for the most appropriate data streams to become available.

A note on the metaverse

I’ve chosen not to include the metaverse as an application in and of itself, the argument being that such spaces will succeed or fail based on their ability to address one or more specific needs already covered here—whether community, communication, or commerce.

What’s missing

In this analysis, I’ve stuck to the technical and design requirements for various VR applications — the main factors that determine whether it’s even possible to build compelling products in each area. But there are plenty of other issues at hand in VR’s development into a new computing platform. A few key aspects that spring to mind:

• Middleware, engines, distribution platforms, authoring tools. These behind-the-scenes pieces will also be crucial in bringing many of these applications to market. And, of course, many of them represent significant near-term opportunities in and of themselves.
• Market sizing, monetization, business model. While I think that all of these applications represent broad markets, some are surely much bigger than others and some will have clearer paths to monetization. Needless to say, the business you’d build around real estate walkthroughs is quite different from one based on live sports streaming.
• Consumer readiness and headset penetration. Some applications require widespread consumer adoption before they are viable, while others can get started just fine on the backs of early adopters or alternative means of providing gear.

Anyway, I’m really looking forward to comments and feedback. What did I miss (maybe shopping)? What did I get wrong?