How Virtual Reality is Changing Storytelling Forever
And why this is the best thing that could happen to cinema
I took the VR headset off. My clothes were ruffled, my hair disheveled, my mind blown.
It was 2014 and I was attending a Game of Thrones exhibit. Facebook hadn’t acquired Oculus yet — VR was little known outside of nerd zones. The event culminated with a virtual trip in the elevator taking Jon Snow atop The Wall —and a surprise freefall. With a (pixelated) Oculus Rift DK1 strapped to my face and headphones blocking the real world, it only took 90 seconds to disorient me.
This was my first VR experience — it was madness. I got a taste of what entertainment would look like in 2020.
For traditional cinema, winter is coming.
Cinematic Paradigms are Shifting
Atmosphere now prevails over the narrative
While I was testing another VR experiment, I witnessed a couple arguing in a Paris garden. But I couldn’t care less about the drama, so I turned away to stroll at the peaceful environment: sunlight reflecting on the grass, passersby, and foamy clouds.
I had spent minutes contemplating my new surroundings, when I realized this could be the experience. I didn’t need to listen to the couple — I was living their moment. The atmosphere had taken over this story.
The first rule of VR for movie makers? Think out of the frame: you can no longer bypass the creation of a compelling universe.
The frame is dead
In the old world of cinema, the eye was captive of a director’s will. The camera told where to look at, and everything outside of the frame was wiped out.
But in VR, the field of view is boundless. This poses a challenge for storytellers: how do you guide the viewer’s gaze towards the action? As VR lets you wander freely, it’s easy to miss crucial plot points. The storyteller must guide you with subtle cues, while respecting your freedom.
So how do you capture the gaze in VR?
One trick used in the short VR film “1, 2, 3 Soleil” was to place the 360 camera rig in a corner, hence narrowing the field of view. Instead of forcing you to stare, the director was tipping you in the right direction.
Coming up with those cues is no small feat: with an additional dimension in VR and a moving baseline, the old rules of composition no longer apply.
How would Wes Anderson render his meticulously symmetrical scenery in VR? He could use anamorphosis, or some Monument-Valley-like perspective mindfuck mastery.
“One of the main challenges for storytellers is learning to think in terms of spheres instead of rectangles”
~ Robert Stromberg (Oscar-winner for Avatar’s visual effects)
Rectangular thinking, however, is counter-intuitive. Unlike spheres, the rectangle is rare in nature. It appears natural because we were taught to think inside a 2D square — but it’s completely artificial.
We experience the world in spheres. Once movie makers learn to break the frame, bringing it back will be hard.
Other artistic disciplines already went beyond the framed picture. Think of Yayoi Kusama with mirrors and infinity rooms, or Sleep No More which adapts Shakespeare’s Macbeth into interactive theater. VR appears to be a natural extension of this.
Yet VR is barely learning to walk
Today, the conditions for a VR take off are optimal. The latest technological advances have made it cheap to watch (a Cardboard costs $15), and accessible to make.
But this goes hand in hand with the proliferation of crappy VR. Most of today’s VR movies are terrible, because having the right technology doesn’t mean knowing what to do with it. Technology has leapt forward too fast for storytellers to catch up. The language of VR has yet to be written.
Moving shots make or break
An obvious challenge to VR is motion sickness. Moving shots are a VR maker’s baptism of fire: the good is insane and the average is appalling.
Trust me, you don’t want to see shaky-cam horror films in VR. Even a simple driving scene can be nauseous if movements are rough or unexpected. Filmmakers are still approaching this challenge through trial and error.
Yet when well done, moving scenes bring to VR its magic and transport you with delight. ‘The Displaced’ documentary opens with a child rowing a canoe, metaphorically taking you to their world.
In the experimental short ‘Anéchoïde’, camera movements are slow and composed. Whether you climb a mysterious spiral staircase or levitate above the roofs of Paris, your mind has enough time to pace itself.
As VR becomes mainstream, it’s fair to wonder if the brain’s plasticity will make us immune to motion sickness: this would become less of a challenge for storytellers. Until then, shaping a comfortable experience remains a major test.
The sound remains a challenge
VR isn’t only about vision, it’s a complete experience that includes sound. To enable a true immersion, hearing requires a 3D sound treatment as well. When you risk missing a pivotal moment by looking away, a loud noise instinctively guides you back towards the action.
That’s why acoustic engineering tricks you into thinking each noise comes from a different direction — out of your headphones.
But the technology isn’t fully there yet — and here’s what I noticed:
- When the sound comes from the front: it feels like it’s inside your head — not the desired effect 😞
- From the back: the brain can source it from the right direction 😃
Not only specific hardware (both for recording and rendering) is required, but each sound also has to be to be mapped with its corresponding object in post-production.
Editing is a nightmare
Traditional movie makers have developed many techniques to transition in between scenes, which we don’t even notice anymore. Furthermore, transitions set the pace of the story — slow and steady, or tense and jerky.
But, in the real world, there’s no curtain call — then how do we switch from scene to scene in VR?
Currently most VR experiences either chose to eliminate the problem with an abrupt ‘fade to black’ effect, or try too hard with artifices. For instance, a stranger’s hands covering your face, telling you to close eyes, and when you pop them open, woo! you’re in the next scene. Very convincing. NOT.
That’s not all, hide the seams when stitching shots together is another headache. When looking up or down, there’s often a blur or some distortion about where the tripod was. Yet 6 (up to 16) GoPros mounted together still offer a higher image quality than a 360 equipment which captures spherical footage from the get-go.
Even if it gets easier to make VR, the final result unfortunately depends on the quality of the hardware. It’s hard to do good stuff with bad tools, and VR bootstrapping doesn’t cut it yet.
Inhabiting the main character doesn’t work
A story is nothing without characters. You’d imagine VR is the perfect medium to put yourself in their shoes — but not literally.
In my experience, being assigned a character’s body was awkward. You know what your body looks like, how it responds to stimuli, and that sense can’t be fooled yet. That’s why I doubt VR porn will succeed in the short-term (although hordes of wild fanboys are trying to prove me wrong here (NSFW-ish)).
Gaming took a stab at it by removing identifiers (age, gender, skin tone) with gloved hands. But this is less of an option for storytellers, who cannot create a memorable character without an identity.
Still, you can’t force someone to slide into a character’s skin. To make that connection happen, there needs to be common grounds between you and the character, and the freedom to extend ourselves to others.
Movie makers, please don’t give me hands that are not mine. Instead, allow me to be whoever I want, to make my own mistakes. Let us enter the scene as weightless creatures, our brains will bridge the gap.
Embrace VR, it’s the future
There’s no adventure as crazy as VR
We have been watching movies for so long that we take it for granted. But you have no idea what’s coming at you with VR: for the first time, you’ll be able to feel movies.
Take close-up shots, the standard way to zoom on a character’s emotions. With VR, close-ups gain the impact of a punch in the stomach. Phantom pain burnt my temples when a junkie pointed his gun to my skull.
A strange girl leaned closer to my face: I could already imagine her plant a kiss upon my lips, yet she aimed for my ear and whispered a cheeky secret. I swear I could feel her breath down my neck.
These shots worked incredibly well: as objects detach themselves from the flat screen, your brain is tricked into feeling the surrounding sensations. Then you dive into the story full-speed.
Blurring boundaries between the movie and the game
With VR you can do more than just enjoying the ride, you can now take the reins. Your eyes become a cursor, the virtual environment can seamlessly adapt to your glance. It could start with tiny, interactive details building up the atmosphere, such as extras smiling back at you. But movie makers can push it to the next level by letting the story evolve with your choices.
Video games have been taking a cinematographic approach in terms of graphic design and flow, even now capturing Hollywood budgets (Grand Theft Auto, for instance). It would only make sense that cinema finally takes a reciprocal turn and gets gamified.
Enough of being spoon-fed scenarios, time has come for the audience to be empowered and make its own.
An inward journey to empathy
“What is real? How do you define ‘real’? If you are talking about what you can feel, what you can smell, what you can taste and see, then ‘real’ is simply electrical signals interpreted by your brain.”
Virtual reality alters our feelings; I experienced it first-hand. We can put this power to good use by building empathy.
Some are already doing it. The New York Times produced a serie of 360 VR videos, on reports ranging from astronomy to war. I dare you to watch ‘The Displaced’ without getting your guts in a knot. It moved me without a drop of blood: instead of telling me how to feel, it gave me the keys to understand a problem, live it with the protagonists, and draw my own conclusions. Hands-on learning goes further than school lectures.
Doom-mongers will claim ‘Technology is making us antisocial! VR will withdraw us from the world!’, but it’s a shortcut. Quite the contrary, VR has the power to connect us together.
“You have to let it all go, Neo. Fear, doubt, and disbelief. Free your mind.”
