Audio Critical for Designing Successful VR
Creating rich experiences in cinematic VR, Gaming and WebVR.
With so many new developments in the field of VR, it’s so exciting to see the hardware becoming faster and cheaper as well as developers creating amazing content. At its core, VR is meant be an immersive experience that puts the user into an alternate reality so, naturally, we initially think about what we’ll see when we strap on the headset. Will it be true to life? A faraway planet? An abstracted reality? Hold up. In order for VR to be truly immersive, we need to put aside visuals for a second and let audio drive the experience.
Let’s say that the three most important components of VR are visuals, interaction, and audio. When creating VR, designing a rich audio experience should be paramount to creating the visuals. Even the best visuals can’t make you feel immersed without the right audio cues to match. My argument for this is that consideration for what sounds are heard and how they are placed in a spatial environment have more potential to trick the brain than visuals have the ability to at the time. In a way, it’s the same idea as if you take off your headphones during the climax of a thriller or horror movie — you lose all emotional investment in the scene and it’s just not scary anymore. The other component, interaction, is also crucial to a complete experience, but I consider it auxiliary. Whereas purely audio or purely visual VR experiences can stand on their own, interaction requires at least one of two in order to be successful (prove me wrong!). However I’ll be the first to say that thoughtful uses of motion controllers and haptic feedback have led to some of my favorite VR experiences.
A brief history
The first recorded instance of manufactured spatial audio was during the film Fantasia in 1940. Using an array of 52 speakers within the theater, three separate audio channels created the illusion of the bumblebee circling the audience. Of course, this eventually led to the creation of surround sound which, in a typical consumer setup (sup 5.1), uses speakers in front, behind, and to the left and right of the viewer to simulate where a sound is intended to originate. Meanwhile, there was also this type of audio recording called binaural which used dummy heads or an equivalent fitted with two microphones to record to a left and right audio channel. The recording could be played back through headphones, giving the listener the effect of being at a concert or inside a radio drama for example. More commonly today, you might be familiar with binaural audio, or 360 audio, if you are a subscriber to the ASMR craze or have seen the (semi) classic Virtual Barber Shop video.
However, this type of binaural audio is not suitable for VR because it doesn’t allow the listener to change position independent of the audio source. What is suitable for VR is a surround sound technique called ambisonics that was only used among a niche groups of audio engineers, sound designers, and composers up until recently. Rather than a stereo audio recording, ambisonics is a full-sphere recording technique that uses an array of microphones to record from nearly all directions. This audio technique creates a speaker-independent representation of a sound field called B-format. Ambisonics have recently become of larger interest because of the use cases it provides in VR applications. Audio in B-format can be decoded by VR software in realtime to match the direction of the video. With ambisonic microphones becoming the standard for VR audio, more and more companies are beginning to produce them.
If you’re not interested in cinematic VR, even better. It is so easy to implement spatial audio in your VR game. In Unity, there is a built-in menu for 3D sound in the audio source component. Link a sound file to an object, slide the spatial blend slider to 3D, and software does it all for you. It will pan the sound between the left and the right channel based on your position to mimic what an ambisonic recording would give you. This past semester I experimented with this effect using the stems from a couple of songs (tracks split into their musical elements). You could virtually move into the middle of the song from the camera’s perspective and hear the instruments around you, changing audio position and volume based on your distance to the object and location of your gaze.
A-Frame is another tool I used to experiment with. It’s a great javascript library for WebVR that has some powerful features. I tried to replicate my Unity experiment in A-Frame, but had issues with loading and syncing the audio files so I changed it up a bit. I made a simple array of floating spheres that would play a sound upon gaze and click. Unlike Unity, there is no 3D sound setting, so all you have to do is give the sound, or object the sound is linked to, a position in which to play from.
Tip: If you want to make your A-frame audio to work from your iOS device, load your project on Safari and then save that URL to your home screen. Should work from there.
Closing thoughts
We all make powerful associations with sounds. They can transport us back to a memory in our lives and affect us at a subconscious level. However, it’s use in immersive experiences is often seen as dependent to the visuals.
“The truth is, for me, it’s obvious that 70, 80 percent of a movie is sound,” he says. “You don’t realize it because you can’t see it.” — Danny Boyle
In no way is audio for VR perfect, but there’s absolutely no excuse to skimp on it with all of the resources available to make it a convincing experience.
Finally, here’s a creative, audio-driven VR experience from Pablo Perez at ECAL that I wish I had come up with.
—
Some VR audio resources and cool videos on sound:
https://www.sonicumbrella.com/
https://facebook360.fb.com/spatial-workstation/
https://www.youtube.com/watch?v=dNJdJIwCF_Y
https://www.ted.com/talks/tasos_frantzolas_everything_you_hear_on_film_is_a_lie
https://www.ted.com/talks/ryan_holladay_to_hear_this_music_you_have_to_be_there_literally
https://www.youtube.com/watch?v=Yd5i7TlpzCk
++++++++++++++++
Get fresh insights, news and analysis with CinematicVR’s newsletter, delivered to your inbox with l♥ve, every 2 weeks. Sign up now!
++++++++++++++++
