Movies in VR: Why they’ll never work, and what will instead.

The stories we tell.

What’s in a game?

I’m a gamer. There are different kinds of games but many — particularly in the big-budget ‘AAA’ category — explicitly seek to tell a story and elicit an emotional response from the player. Trigger points, scripted events, voice-over narration, pre-rendered cinematics… all manner of mechanisms to try to railroad the player into a pre-determined narrative and bludgeon an emotional response into the user. What all these mechanisms do is momentarily interrupt interactivity. Regardless of what the player was doing, thinking, planning, or experiencing internally: They now have no choice but to look in this direction at this specific event that supposedly has narrative meaning and emotional resonance for the hero they are playing.

It always* falls flat.

If you’re interested in why this one-way telegraphing of story doesn’t work in a categorically interactive (read: two-way) medium, you can go read about Ludonarrative Dissonance. Here I am more interested in what does work. And yes we are going to get to VR movies — just bear with me.

In all my countless hours of gaming, there have been a few “stories” that really did impact me, heart mind and soul, stories that I remember long after the game ends. What is interesting is that none of them were explicitly planned by the games’ designers — although they were explicitly facilitated, perhaps even incentivized, by the designers.

Everybody loves examples:

Most powerful for me personally are the stories that emerge from “4x games” such as Civilization, Crusader Kings, or even SimCity, where you are thrust onto an empty map and told to build. Through hundreds of decisions and events, you shape the course of your little empire and the map-world at large. Cities rise and fall; regions change hands; alliances signed and broken.

Say there is such-and-such a village that I settled way back (read: hours ago) in the early game, and now it’s critical to my economy because it is next to such-and-such randomly-placed strategic resource, and I have twice lost said settlement to such-and-such belligerent neighbour, with whom I always had peaceful relations until they developed slightly more advanced weapons and swarmed my undefended border, seized the city and razed the temple, and twice I have mounted huge hour-long efforts to win it back… You better believe that the feelings I have about that village and towards that rival empire, even though both entities were generated at runtime by a host of randomized variables, are far stronger and deeper than anything I have felt towards (for example) Elizabeth in Bioshock Infinite. Moreover and more interestingly, I feel like there was a real story to that city.

A team and role-based FPS like Team Fortress 2, more often than not degenerates into a mosh-pit. But every once in a while, with no explicit leadership or communication, the slime mould of anonymous avatars self-organizes into a super-organism: Coalescing around a clear strategy; replete with coordinated lines of defense; forward operating bases maintained by engineers, served by an improvised transit-system of teleporters; a reliable healthcare system of medics; etc. It’s a marvel to behold. And there’s a story in that. A real story the way a great baseball game can be a story: There were highs and lows; moments when all seemed lost; and heroes who distinguished themselves through glorious feats of leadership and cooperation. For another brilliant and very different example of emergent social narratives in non-communicative (non-verbal) online games, look to Journey.

I would also like to mention open-ended adventure games like Myst and Cradle, where there is a narrative implanted by the designers but presented almost just as context or justification for the gameworld. One can engage with it or not — it doesn’t interrupt the interaction mechanic, it’s just there like background music. It is an “ambient story.” The real narrative experience that I lived and remember is one of awe, exploration, and discovery in a strange and wondrous alternate universe (and the occasional hard-won puzzle solution). A traditional big ‘S’ story? Technically no. But it certainly evoked the emotional resonance and memorable engagement that every storyteller seeks to achieve.

A pattern emerges:

What’s common among these anecdotes is that the “story” that impacted and stuck with me was always a product of, never an interruption to, the interactive mechanics of the medium. It emerged organically from all the myriad procedural systems at interplay, and I was integrally part of it. As much as I had an impact on it: vice versa. Basically:

It felt like a real experience; and in many meaningful ways, it was.

The pattern here is clear: Emotional engagement in interactive media is a direct product of the interactive processes themselves, not an ancillary artifact that is irrelevant or counter to the mechanic of interaction. Marshall McLuhan would be proud.

*Of course some folks will cry foul and point to examples in which a designer-directed narrative was successfully engaging in a first-person linear game experience. Last of Us, Walking Dead, and Firewatch all come to mind. But as I am playing Firewatch last night (really a beautiful product, you should check it out) I couldn’t help but think that it would work equally well as a movie. What does this story get out of being interactive, other than taking longer to get to its conclusion? If in a game the linear (or finite branching) story works really well, it is only because it is a great story to begin with. All other aspects (interface, art direction, voice acting) when well executed work to support it. But every time any person plays through Firewatch, they will be getting more or less the exact narrative experience that the designer intended. It is nominally “interactive,” but not really. It is still a movie (or novel) shoehorned into a game engine. It does not fully manifest the characteristics of the medium in which it is transmitted, and most of the time it fails. (If I’m still hearing screams that Last of Us could never work as a movie and needs player choice to motivate the story, please show me a choose-your-own-adventure novel that was ever a better story experience than a comparable normal novel. Ever. Anywhere. Please.)

Wrong, wrong. It’s all wrong.

Ok so VR then. Much talk is circulating about how a new visual language will have to be developed for VR, because the old directorial + editorial tricks employed in Cinema to direct viewers’ attention and imply meaning will no longer apply. Some are hearkening back to Theatre, where the audience is (more or less) free to look where they please, and their attention is directed through lighting and audio cues. But VR and Theatre are apples and oranges. In VR there is no audience “here” and stage “there” (Theatre-in-the-Round notwithstanding). The “fourth wall” has been moved behind the camera, and in a few hardware generations all four walls will become immaterial (check out my post on “Adaptive Virtual Environments”). Nor has the viewer become the camera. The camera was a window onto a story universe, but VR fundamentally breaks that window. We need to think in different terms.

“How do we direct the viewer’s attention in VR” seeks an answer to the wrong question.

Riddle me this: How does an architect “direct a viewer’s attention?” Or for that matter a city planner? Or a festival/event planner? A social app developer? But wait: Do narratives take place in cities or at events or on social media? Hell yes they do. Did the designers explicitly write those stories and force users to pay attention? Hell no they didn’t. What they did do was use every tool in their shed to facilitate emergent narratives within their environments, whether physical, temporal (in the case of festivals/events), virtual, or networked. They carefully distribute regions of density and interest; friction and fluidity to incentivize particular flows of traffic; chronology and proximity to suggest association and meaning. Through tricks and choices unique to each discipline, they can even influence certain experiential outcomes (for example choices of colours, zoning, music, food, homepage shortcuts, font sizes).

Open the kitchen onto the patio and you will optimize for different narrative outcomes than making it a windowless nook by the TV den.

Storytellers in VR need to think in the same terms. “Stories” in VR will no longer be strictly linear. Neither will they be finite branching. The 2016 Tribeca Film Festival used the term “Storyscapes,” a term which I find captures the new paradigm quite well. A storyscape is similar to the aforementioned concept of an “ambient story.” There’s something happening. You can engage with it or not — it’s going to happen anyways. Only here it’s happening independent of a camera’s, or stage’s, or storytellers’ POV. It is happening all around you. You are in a story.

This is why traditional filmmakers are having such a hard go at it. The old tropes of sequential plot, chronology, associative montage…you can just throw all that right out the window. VR is about immediacy and immersion, not about building suspense through a slow-drip of information or precisely-timed cuts.

Just like in every new medium, certain types of narratives will be better suited to its language than others. The epic poem, opera, novel, radio play, feature film, and television serial all had a particular type of story that worked best for that medium. The types of stories that will work best in VR - at least in the early days - will be those that emphasize a sense of place: Really doubling down on setting and environment to convey emotion. Hence why I like the term storyscape. The setting isn’t just decor to hang off the story, the setting is the story.

But as more and more naturalistic (read: intuitive and un-abstracted) mechanics of interactivity become available to the technology — both “physical” interactivity through full body motion tracking as well as verbal interactivity through AI conversation engines developed for “bots” — VR will really start to come into its own. Successful VR stories/experiences will come to reveal more closely the impactful game experiences I outlined at the start of this article. They will also be accessed by a wider and more mainstream audience once the interaction learning-curve is demolished. Information will flow two-ways, spectators will become players (in the Shakespearean sense). And here is where it really starts to get interesting. Because these will be the new categories of experience that were never possible before VR.

Janet Murray stated that the computer is fundamentally and uniquely: “procedural, participatory, encyclopedic, and spatial.” (I can’t help but think that would make an apt tagline for No Man’s Sky). She also pointed out, in concordance with a famous tenet of Marshall McLuhan’s (the forms of all preceding media serve as the content of a new medium), that all content from the Dead Sea Scrolls to an episode of Law and Order can be easily expressed within a computer. It will be in combining and manipulating the content of the old media with the unique properties of the new medium that we will begin moving beyond the era of the photoplay in VR.

On a theoretical level, it’s deceptively simple: Take a movie, and all its elements: characters, acting, art direction, music, sound design, etc. and add these four properties: Procedural (generation based on pre-determined rules and run-time inputs); Participatory (ontologically interactive — if a tree falls in the forest…); Encyclopedic (a built-in background matrix of information/eventualities available to explore); and Spatial (breaking free of the two-dimensional screen space).

For Your Consideration:

Want to show what it’s like being a Mexican immigrant getting pulled over on the interstate? In a movie, it would be this one particular Mexican immigrant in this one precise moment, and then this exact thing happened to this precise character, followed by this exact event, etc. A skilled filmmaker could build empathy in the audience for that character, but the audience remained passive and distant spectators of someone else’s experience. Now consider this:

You are a Mexican immigrant. Not this one or that one; it is you yourself, now wearing this skin, sitting in this car, with a Texas state troopers’ lights flashing in the mirror. What is going to happen? Well it depends on what you do. The writer doesn’t even know. Go ahead, try something. The story will react according to the writer’s general parameters (i.e. the officer’s personality; the paper bag in your glove compartment), but the plot specifics will be determined at “runtime.”

Which do you think will be the more memorable, powerful experience? The movie situation that you observed as an omniscient spectator? Or the VR storyscape that you lived: That unfolded partly according to your actions, and that exposed you personally to the outcomes?

The stunning conclusion (finally)

So there you have it. That is what will work in VR “storytelling.” You can forget about movies with 360° cameras/screens. That is dumb, and manages to miss out on everything that works great in Film as well as in Virtual-Reality. We will probably have to come up with a new word for movies/stories that are procedural, participatory, encyclopedic and spatial — probably even a newer word than “storyscape” (or Janet Murray’s verbal faceplant “Cyberdrama”). But as in everything, the journey will define the destination, and there will be plenty of missteps along the way.

Since the dawn of humanity, storytelling has been the engine of empathy in society. And if VR is going to change storytelling, VR is going to change humanity.