HOW should a story be told in VR?
A story has to be adapted to the medium it is to be consumed in. Books are adapted to movies, screenplays are adapted to books or plays. Each medium has its characteristics — its strengths and weaknesses.
What are the key elements that define a movie experience in VR? How should a story be told so as to make it a compelling VR experience?
In traditional movies, the entire story is experienced through the frame —
the window through which the director of the film presents visuals, sound, and dialogue in a manner calculated to evoke certain emotions from the viewer, depending on the story and the scene.
In a book, a writer evokes those same emotions through words, using tools like scene descriptions, action descriptions, clever dialogue, inner thoughts of characters, key turns of phrase, and vocabulary that triggers responses from readers.
Even if the story being told is the same, a writer adapting a book to a screenplay chooses to highlight certain events from the book and discard others, often to the irritation of readers who love those events in the books. For instance, conversations with descriptions of inner thoughts in the characters could be shortened to a brief exchange, with the emotions depicted in the inner thoughts conveyed by a look or gesture by a skillful actor. On the other hand, a chase scene, or a physical conflict could be lengthened far more than it was in the book, with the intent of enhancing visual drama.
Different mediums have different advantages, and the same story needs to be presented differently in each medium so as to fully utilize the potential of each.
So what are the key elements that define a VR experience?
Immersion
VR is the first technology where a viewer putting on a headset with the right specs (field of view, resolution, pixel persistence, refresh rate, etc.) can actually believe that they are in a different world. The technology has a ways to go before this illusion is seamless and sustainable without causing VR sickness, but once achieved, complete and believable immersion will be a minimum requirement of any VR experience. The illusion is achieved both through visuals that track the user with millimeter accuracy, but also with binaural sound that is heard from the right direction and with the right volume to create an illusion of a physical space that has depth. Future VR systems could also use special bodysuits, harnesses, and special seats that use haptic and force feedback to engage tactile senses and further deepen the sense of immersion.
Interactivity
A virtual environment, no matter how seamless, will never be truly believable unless the user can interact with the environment in a meaningful way. This means that a viewer in a virtual environment should be able to interact with objects in the environment as naturally as they would interact with objects in the real world, and those interactions must result in responses in keeping with the rules of the virtual world.
People communicate with their voices, their eyes, their hands, and their body language. Communication can be verbal or nonverbal, intentional or accidental. Additionally, computer systems can also draw information from signals that other humans cannot easily detect, such as temperature, blood pressure, heart rate, brain activity, etc. As the technology improves and sensors (bio, visual and other kinds) get more ubiquitous, it will become possible to capture most signals that humans generate today and use the information to guide the virtual experience for that viewer.
After that extended introduction, let’s come back to the original question —
How should a story be told so as to make it a compelling VR experience?
Maybe it’s best to explore this by writing a sample VR script in different genres. I'll keep the storylines for each fairly generic and emphasize points (in italics) that a VR experience would do differently.
A Love Story
This is a tricky genre to start out with, given the nuances, the subtlety, and the subtext that goes into a good romance, but let’s just dive into the deep end, shall we?
The classic act 1 setup — boy meets girl, girl meets boy. They flirt. A flush of attraction. Verbal sparring. A first date at a coffee shop. He’s nervous, she’s amused and yet attracted to him.
It’s a simple scene, albeit a little clichéd — let’s see how we can make it fun.
EXT. SUMMER BBQ PARTY — LATE AFTERNOON
The scene is filmed using live actors in a real location. Scenes are filmed in two versions for each “main” character. A main character is any character that the viewer can choose to embody in the first person. One version of the scene is filmed from the point of view of the character and one is filmed from two third-person perspectives — one from front and right and the other from behind and right.
Viewer starts the scene as Ben in the first person. Viewer has the capability to switch between first and third person perspective at any time during the scene.
Dialogue is spoken by the live actors in the scene.
BEN walks around the yard, nodding hellos at people he knows, looking for the host JACK. He’s down a couple beers, and the lazy summer sun feels good. He looks up to the deck of the house and sees… SAM.
BEN : “Whoa… stop the presses!”
She’s beautiful. Jack walks up alongside Ben, catches him staring up at Sam.
JACK : “Out of your league, boy-o. Way out.”
Ben can't help but feel Jack is right. But he can't stop staring. Sam catches him looking up at her. She smiles. Ben can’t believe his luck.
Viewer has the option to continue as Ben’s POV, switch to Sam’s POV, or watch the scene from a third-person perspective.
Ben walks up to the deck. Sam turns around to meet him.
Ben : “Hey… I'm… Ben.”
Sam : “Sam. Nice to meet you. How do you know Jack?”
Ben : “We were roommates in college. You?”
Sam : “I am his pot dealer.”
Ben : “I served with Jack’s dealer. I knew Jack’s dealer. Jack’s dealer was a friend of mine. Senator, you're not Jack’s dealer.”
… and so the scene progresses. The scene above describes limited interactivity for the viewers, but the viewer actually has a lot of choices to interact within the scene. They can switch between first and third person perspectives for the main characters — maybe the first time, they experience the scene passively, just letting it play out. They like Ben as a character, so the next time, they view it from Ben’s POV as he sees Sam for the first time and goes up to talk to her. During the scene, Sam runs a finger down Ben’s chest in a flirtatious gesture. The viewer wearing a haptic bodysuit could experience the touch as Ben would have in the scene. For a viewer already engaged with the characters, this would create an extraordinary moment of immersion in the VRX.
Next scene:
INT. COFFEE SHOP — DAY
Ben and Sam are on their first date. Conversation is flowing, and Ben is having fun talking to someone who’s provocative, witty, and self-assured. He’s attracted to Sam, and wants desperately to impress her.
For her part, Sam is enjoying the date. She sees that she’s making Ben a little nervous, so she starts a little more subtle flirting just to see how rattled he gets.
Many interactive touches could be added here. If the viewer is in Sam’s POV, and Sam brushes her leg against Ben, she could feel that brush. A viewer in Ben’s POV would feel the same gesture.
Viewers could also be allowed the option to initiate gestures that feel natural and fit within the constraints of the scene without interrupting the flow of the scene and the dialogue.
Even a scene as simple as this is extremely difficult to get right. Enabling the right moments of interactivity in order to preserve the sense of immersion will take a lot of careful planning, precise filming and some clever software programming to tie it all together. This is especially hard to do with live actors who would have to perform multiple versions of the scene with slight differences to accommodate for gestures initiated by the viewer. This is in addition to the versions needed for third person perspectives.
It’s definitely a lot easier to implement this interactivity using digital avatars in a virtual environment. Avatars do not need to be filmed ahead of time, and can react in real-time to user signals.
There have already been some experiments done on how this could work — for example, Tore Knabe’s “Coffee without Words”, where an digital character moves her eyes to follow the player and meets your gaze if you look at her.
However, the downside of digital avatars is that they still continue to fall into the “uncanny valley”. Also, the subtlety of human expressions and body language that is required for believable flirting is very hard to code for.
The purpose of my proposed sample scene is to create a vision of what a truly engaging VR experience could be. The technology to realize this vision does not fully exist today, but I believe it’s only a matter of time.
References:
- Tore Knabe’s experiment for switching between first and third person in VR.
- Minority Media’s VR game “Cali”, which allows a player to switch between first and third person and includes first person sequences where the player can use controls to flirt with the object of desire, Cali. They might choose to caress her face or kiss her, and she will react.
If you enjoyed this post, please do recommend it. Please also take a look at some of my other articles on the topic:
- Simple techniques to enhance “presence”
- These techniques applied to scenes in the 2010 movie, “Inception”
You can view all of them at my publication : Storytelling in VR
