The Language of Presence: A Virtual Reality Glossary for Storytellers, Producers, and Viewers
Virtual reality is one of the fastest-growing media of our time. The rate of development of the various technologies that make it possible is breathtaking, with new advances being announced almost daily: new ways of capturing data, of allowing users to interact with content, of mixing and merging formats, of distributing the results. News organizations have launched into creating VR content, most notably The New York Times, which attached a million Google Cardboards to its Sunday paper and has made its VR app easily available on smartphones. More recently, Time Inc. has jumped into the fray, kickstarting its new LIFE VR app with a Sports Illustrated piece on Mount Everest.
But the very factor that makes VR so thrilling — this stratospheric rate of change, bringing with it endless new possibilities for creative expression — also makes it dauntingly complex and difficult to explain, let alone manage. The range of experiences available to consumers is vast: from videos that simply offer a spherical field of view to interactive environments that deploy the artistry and techniques of video games in the service of journalism.
Actually producing VR is an even more complex and fraught process. Recommendations for the best 360 camera rig can become outdated in a matter of months, making hardware decisions extremely difficult. Volumetric video recording can sometimes replace an expensive motion capture session — but it takes an experienced, well-informed producer to make the call. The kind of headset that the viewer will use determines the overall parameters of any given piece, meaning even the earliest ideation session must be informed by an awareness of the capabilities of that particular platform.
Part of the problem is that even the terminology we use is undefined and in flux. It’s one thing to try to explain to someone what “volumetric” VR is like. But perhaps the preferred term should be “room scale” or “walk around”? How about “volumetric video” versus “videogrammetry”? Or even “cinematic” versus “kinematic” VR?
In a very simple step toward making things a little more understandable, we’ve taken a first stab at some key terms that define the parameters of the field. We’re a long way from a uniform, integrated production and distribution ecosystem; but if we can begin to craft a shared language that lets us agree on what it is we’re trying to achieve, that feels like a step in the right direction. So: Drumroll, please …
Presence
This is the single defining characteristic of virtual reality: the way in which, thanks to a certain combination of sensory input, your mind can trick your body into feeling as though it is somewhere else. Though “presence” has become the most overused, clichéd term in the field, the phenomenon is not a fad, a novelty or a gimmick. It is an established field of neuroscience, with both its positive and negative potential debated by academics around the world. The various factors that affect the level of presence achieved — from visual detail to frame rate to the realism of a character’s body language — have been tracked here. There have also been warnings such as this one about the dangers of a medium that can tap so directly into our consciousness.
Empathy
Another overused term, and another well-documented phenomenon. Feeling present in a scene generates empathy on the part of the user toward the characters depicted: Viewers have a much stronger emotional response compared to watching the scene on a 2D screen.
360 video or cinematic VR
Video capture that creates a spherical field of vision, allowing the user to look in any direction as if they were at the center of a globe. Although this creates a certain degree of immersion, the viewer is tethered in that central spot; they cannot alter their position within the environment. 360 video is the most accessible of all types of VR. In its simplest form, it is viewable using only a mobile phone: By either swiping on the image to change orientation or waving the phone around in the air (“magic window”), the user can take in the entire field of view.
Headsets can up the level of presence. Google Cardboard increases the sense of immersion; Samsung Gear VR has better optics and incorporates basic built-in user controls; Google Daydream comes with a hand controller that enables more elaborate interaction.
Volumetric VR, aka room-scale or walk-around or kinematic VR
Any kind of experience in which the user can move around inside the environment, examining the scene from different viewpoints and observing characters from different angles, etc. This requires actual space in which the user can roam (hence “room scale”) as well as external sensors that track the position of the headset, allowing it to adjust what the user sees in real time based on their position. As the technology develops, full motion tracking will be possible via cameras mounted on the headset, obviating the need for separate external sensors.
Traditionally, the only means of creating walk-around environments has been via video game platforms such as Unity and Unreal. Since these environments have been built using computer graphics (CG), they have not attained the same level of realism as 360 video. Hence the trade-off: visual verisimilitude versus the ability to move. However, new techniques such as photogrammetry and volumetric video now promise the best of both worlds: fully 3D environments and characters that achieve the same level of visual detail as traditional still images and video footage.
Spatial narrative
Storytelling in which the physical and temporal dynamics of the scene — who moved where, and when, and their relative positions at each moment — are fundamental to fully understanding what happened. Volumetric VR is uniquely able to convey these parameters — the distance between a shooter and his victim, for example, or the exact sightlines afforded by a particular vantage point.
Photogrammetry
A means of capturing 3D spaces in high-resolution photographic detail. The photographer takes multiple images from multiple points within the environment; a postproduction process triangulates each of those images relative to each other, creating a geometrically precise “mesh” onto which the images are mapped. The result is a virtual environment in which the user can walk around, captured at a level of detail that rivals still photography.
Volumetric video, aka videogrammetry
A variant of photogrammetry in which people are recorded by an array of multiple cameras, usually in a dedicated lab. The result is a video-quality 3D figure that can be viewed from any angle, even as it moves. The potential for placing these characters into a volumetric environment opens up huge possibilities for the level of realism achievable in walk-around VR.
Nausea
One of the most common misconceptions about VR is that the medium itself causes motion sickness. This is generally not the case. While a small percentage of users do have a mild reaction to any kind of immersion in VR, the vast majority of cases are caused by a simple mistake on the part of the director — the use of extended tracking shots. When your mind is immersed in an experience, your body comes along for the ride. Thus, any movement you sense that is not commensurate with the actual movements your “real” body is making will cause an uncomfortable disconnect. In a tracking shot, your eyes will tell you that you’re in motion, though your inner ear informs you that you’re standing (or sitting) stock-still. While not everyone will get ill from these conflicting sensations, a large proportion will, and that has to be balanced with the creative decisions made by the journalist or filmmaker.
Embodiment
The level of immersion achieved in a volumetric piece that allows users to move around inside the scene at will, using their own movements to control their visual orientation. One of the most powerful and obvious examples is any scene that places users on a precipice or above a steep drop — the feeling of vertigo is palpable, and they have to be coaxed to step “off the ledge,” even though they know on some level that there is no actual drop in front of them.
Other terms
Stay tuned for a forthcoming blog post in which we will review other terms like augmented reality (the overlaying of content on top of the reality viewed through a screen) and haptic feedback (the creation of bodily sensations in direct response to content).
