The immersive nature of XR is a challenge to those accustomed to creating narratives in traditional media. Films, theatrical productions and art works are written, directed and created as if the audience is looking through a window from one particular point of view. XR, on the other hand, is a completely 360 experience. Creators must become elaborate world builders, filling a larger space and considering an audience who could be looking anywhere at any given time.
How can XR break away from these traditional writing perspectives and break the boundaries of 2D narrative? How are sensory and synesthetic tools used for storytelling with multi-sensory design? VR narrative goes beyond the hardware and software itself. It extends to the environment and atmosphere where the experience takes place in both the real and virtual world.
Pulling from leading innovators in VR storytelling in 2019, we will explore a 360 experience, an interactive narrative and a live experience that demonstrate the future of storytelling in the immersive medium.
360 VR has been critiqued as mimicking the 2D medium of film. What’s different? However, when 360 VR is used to grant perspective in stunning 360 visuals that accentuates presence in the narrative, it is highly effective.
In Everest- The VR Film Experience, Sherpa Tenji attempts to climb Mt Everest without the use of bottled oxygen. His journey is captured by cameraman Jon Griffith with 8K 3D Ambisonic Audio Virtual Reality. It’s one of the few 360 experiences that truly tempts viewers to look up to wonder at the highest elevations on Earth, and down into the abyss of ice and snow and dangerous crags. By capturing the climbers and the greatness of the mountains in life size- viewers are invited to be a part of the ascent.
Much of this effect would have been lost if not for the great attention to audio. Without life-like clear and spatial audio, 360 video can feel flat and distant. Howling winds, the steady crunch of snow under the climbers’ shoes, strained breath and words of encouragement were so real and near that it was easy to cross arms across one’s body during this experience as if to protect from the frigid winds.
Interactive and Non-linear
Eleven Eleven, an original science fiction immersive narrative by SYFY and NBCUniversal International Networks, is an original multilinear narrative experience designed in every purpose for VR and AR storytelling. It’s a science-fiction narrative with multiple characters who are on a race against the clock as a planet ticks down to its final moments. Inspired by immersive theatre, the experience features multiple storylines running simultaneously on a common timeline- in real-time.
Three distinct modes are available for the audience to have complete agency as an observer. Story Mode enables the viewer to attach to a main character and automatically follow them as a passive audience. Explore Mode is an active role where the audience can freely explore the world to follow secondary characters and search for clues to uncover the mystery of the world’s end. Finally, Goddess Mode empowers the audience to be omnipresent and view the world as a 3D board game of sorts where the entire narrative unfolds in a ‘table top’ version of the story.
With original music by Bleeding Fingers designed as a spatial score, a character’s individual soundtrack changes dynamically based on the users position- seamlessly mixing with an alternate character’s scores as a user navigates between stories.
“In a traditional TV show or film, every viewer sees the same frame, at the same time, every time and the composer creates a musical soundtrack to suit. With Eleven Eleven, since we have no idea where users will be looking at any given moment, it posed a unique challenge when composing the score. We decided to create a “Spatial Score” for each character that fluidly adjusts in volume based on a users’ proximity to that character,” wrote Mehrad Noori, Director, Global Programming, NBCUniversal International Networks and Creator and Executive Producer of Eleven Eleven.
Storytelling With Live VR
The music and tech group Miro Shot is the first to tour a live VR concert, looking at the future of live entertainment with immersive tech. The story? Worldbuilding with future tech. What does VR mean in live performance? The group began staging DIY versions of their VR performances while based in an east London warehouse in 2017. “We’d fill the building with smoke and blend the atmosphere of a gig with the sense of being physically present in the non-physical world that VR gives you,” said Miro Shot’s lead creative Roman Rappak.
In a typical Miro Shot show the audience enter a room filled with projection mapping, dry ice, and atmospheric lighting. Once seated in VR headsets, the audience experience an audiovisual journey of virtual landscapes, live feeds of the concert, wind-fans, haptic seats and custom scents, all soundtracked by the band. Miro Shot is looking at immersive experience beyond the VR headset. Theatrical elements such as smoke, wind, temperature, projection mapping and more are a part of developing a new world for the music to enhance the narrative, both live and virtual.
VR in these shows goes beyond what is seen in the headset. The pass-through camera enables the audience to see each other at times, to be connected to the physical space, and then to experience the world of the music itself, transitioned to a new reality.
Researchers are also paying close attention to VR and music in a live setting. In Kent Bye’s podcast “Neuroscience & VR: Music, Body Sway, & Synchrony at the Institute for Music and the Mind,” he speaks with Laurel Trainor, Director of McMaster Institute for Music and the Mind. Trainor reveals that when people experience a live music concert together, they are more connected to each other as their brains are doing the same thing at the same time. These social cues gives us a sense of belonging and demonstrate the importance of people being physically synchronized with one another for social group dynamics. Music accentuates this feeling as people move to music together. When people move in synchronization- they feel inherently feel more connected.
FoST at VR Days Europe
In the Future of Storytelling, attendees will learn from leading visual storytellers as they showcase their works and explain the processes behind their creation. This year at VR Days join: Joris Weijdom, lecturer at HKU University of the Arts Utrecht and researcher and designer of mixed-reality experiences focusing on interdisciplinary creative processes and performativity;