Making Music for VR

by McKenzie Stubbert, composer for Within & Here Be Dragons

Wright Brothers’ first flight in 1903 (photo credit: U.S. Coast Guard)

The aviation industry took off with the Wright brothers’ Flyer at Kitty Hawk 103 years ago. People had been dreaming about flight since they first noticed birds. The Wright brothers made dreams a reality overnight. Pioneers learned what to do with powered flight once they had it. That’s not the case with Virtual Reality; which includes 360° video and immersive, interactive experiences. To me, VR in its current form is an open-ended artistic medium.

It is also the most exciting experiment I’ve encountered as a composer.

Every other medium I’ve explored — from film to recordings to live performance — has a rich history of traditions made, broken and reinvented by artists that have hammered stakes so deep in the ground they made others either want to work harder or quit. That’s not to say these mediums are finished. Not by a long shot. But VR is an undiscovered country with more road than rules.

We have a sense of its power and its future, but only just that. VR is a medium decades in the making only now available to mass audiences. What we currently call VR will likely change so drastically that I expect we’ll look back at this time and laugh at how hard we tried to flap our wings in order to fly.

People behind the content and technology of VR work hard every day to both improve how it works and develop what it even is. Sound is obviously critical to their work. Hearing an accurate representation of the space helps convince the brain that what is happening is happening. Music serves an altogether different, but vital role.

That’s my sandbox.

Music Guides the VR Viewing Experience

I grew up with a deep love of cinema, obsessing over film music, composers and their relationships with directors. Writing music for a 360 space wasn’t something I sought out as I explored where I wanted to go in my career. What started as a very cool one-off project with a director I greatly admired, Chris Milk, turned into writing music for over 20 VR experiences in a year and a half.

Film scoring is an attempt to stitch human emotions into the image on screen. “Is this a camera moving through the water or a hungry shark?” The same can be said for VR. “Am I looking at an empty field or did something just happen here? If so, what? Where?” The biggest difference is that each ‘moment’ is an experience dictated by the viewer as they are in control of where they look.

As a composer, my job is to help convey the director’s vision of what the viewer’s emotional journey will be. Immersive sound design and foley helps orient the viewer, nudging them to look over their shoulder, like seeing the giant sperm whale pass by in The Click Effect from Sandy Smolan and James Nestor. The music, however, helps the viewer process how to feel about what they are seeing at any given moment and make sense of the images that surround and encompass them. It provides emotional context for what they experience. In this case, a sense of wonder and discovery.

Why VR Needs Music

Sound and picture can establish a literal time and place. Music can establish an emotional time and place. This will likely always be the case.

We currently approach VR films as a relative of cinema, however, the visual rhythm that comes with VR filmmaking is fundamentally different. For live action documentary pieces like “The Displaced,” the editing is much slower with consistently longer shots that fade in and out. They are also journalistic in nature so the amount of emotion we want the viewer to experience is far less specific. We want to suggest a tone and let the viewer bring the rest. For a single sequence experience without characters or distinct narrative like “Evolution of Verse,” the approach is far more like scoring a visual ballet. The emotion conveyed is one the director, Chris, wants you to experience. Both approaches take an incredible amount of work, even if what you end up with (what the viewer sees and hears) is simple and sparse.

As we move forward towards further interactivity, the needs of the score will change. The current approach is either to score set sequences like a film (precisely scored moments) and more interactive sequences like a game (loops that provide general emotional tone). My hope is that we find a new place where the viewer experiences interactivity that feels like it’s being scored as precise as a film.

The VR industry is still in the era equivalent to the nickelodeon period of cinema. Creators are still trying to figure out how, or even if, acting works for the medium. To discover how much interactivity will be possible while constructing a narrative. And as the headsets develop, how long a person will even be able to sit inside an experience.

There’s a wonderful level of experimentation going on in the industry right now. I believe music for VR will continue to be a vital adhesive that ties the elements together while content creators and technologists build the rest of the picture. As Chris discussed during his latest TED Talk (which I arranged and composed the music for while making real-time adjustments to his delivery cadence), VR will shift the way that stories are told — but viewers can expect to rely on music to help guide their way.