Sound is the king
How much audio is worth in VR?
During our adventure with virtual reality we watched hundreds of content and each of them was surprising in a way. Every 360 degrees experience is very personal. The main goal of every producer is to navigate the participant through this new, often still uncovered movie format. These pieces put ourselves inside the film, we can feel part of them. When you watch 360 degrees movies you usually focus on one part of it and sometimes can’t follow the main story. But in addition to the video we have audio which plays a crucial role. VR films which use spatial audio helped us boost emotions.
360 degrees films provide additional stimuli to our brain. We’ve been waiting impatiently for new releases in this market. But discussions on movies we watched revealed that everyone saw a different film. Each of us focused on different parts of the content therefore experienced it very individually. This was encouraging but at the same time we could not probably experience correctly what the producer meant. We missed many important parts of the scenario. This phenomenon was confirmed by pop-up VR cinemas organized by us all over Europe at film festivals where newcomers met with VR veterans (including VR Days, CPH:DOX, Cannes Film Festival, Venice Film Festival, Encounters, Geneva Film Festival, Millennium Docs Against Gravity and many more). Even if the movies were of very high quality we had to explain the participants how to absorb them. Decades of watching standard films limited to the TV screen affect the way we watch and experience multimedia content. At the beginning the participants were watching 360 degrees videos in the same way, staring in one direction. Only after certain introduction they started moving their heads, looking around which helped them to experience the film completely.
VR production goes forward
I remember it very clearly, almost 15 years ago a friend shared with me the Virtual Barber Shop Hair Cut — 3D Sound content. I was struck how you can play with the sound around the participant. Still today it serves as a very interesting experience. Many years passed, we entered a new decade and can experience spatial audio combined with 360 degrees pictures. This was really surprising! Quality of VR content has been increasing, the same has happened with the audio. More efficient VR headsets helped us to use spatial audio in 100 %. It was clear then that we need to include these functionalities in Showtime VR software for easy and synchronized VR playback. It was not easy, but after a few weeks of development we added such features which work on many VR headsets on the market — Oculus Go, Oculus Quest, Pico, Lenovo Mirage, Samsung Gear VR, HTC VIVE Focus, Skyworth and many more. At first the feature worked only with .tbe (Two Big Ears) Facebook360 audio format. Currently it covers Ambix audio embedded in the video file which becomes another standard. Showtime VR was one of the first 360 degrees video players in the market which offered binaural audio features.
Sounds around you
3D audio follows moves of your head which helps to cheat the brain even more accurately. At the beginning VR movies with standard stereo sound did not bring up all advantages of virtual reality. The brain “felt” it was cheated and the experience was limited.
3D audio, in addition to stereo is aimed at copying the natural sound experience around us. Spatial audio engages the participant much more strongly. Sound can easily move around the participant in the same way as they do around us in the real world. We can hear where exactly the sound comes from and how far it is from us.
You can create 3D audio in one of two major ways:
- Record the spatial audio originally using dedicated microphones
- Create spatial audio effect in postproduction by simulating the 3D scene set up
You would need to use advanced software to create spatial audio in postproduction. It takes into account many variables which influence our brain’s perception. They take into account delays, volumes, frequencies or wave bouncing. The end result is comparable with recording with special microphones. The major types of recording spatial audio include: binaural, spatial (ambisonics / ambix and FuMa) and TBE. The most simple way of recording 3D sound is binaural sound. It involves two microphones placed at the ear level. This way enables to reproduce the audio in experience as close to the real experience as possible. Spatial involves sounds flowing around the participant, above and below the head in the 3D space. Spatial is not transmitting sound automatically to loudspeakers. It is based on a B-format standard which decodes audio to an individual loudspeaker set up. Depending on coding type it includes: ambisonics/ambix and FuMa. FuMa used to be more popular until Youtube chose ambix as the major format. You can lean more about spatial audio from VRTL Academy: https://vrtl.academy/news/how-to-record-spatial-sound-in-a-vr-360-film-set/.
TBE format was created by Two Big Ears company as one of binaural sound standards. Facebook bought this technology and implemented in its Facebook360 project.
You can check how binaural sound works:
https://www.youtube.com/watch?v=34Y0dwVBq4c
https://www.youtube.com/watch?v=51za5u3LtEc
Ambisonics B-format:
https://www.youtube.com/watch?v=mcUwiYwgBHw
If you’re looking for a good example The Scream VR app provides and interesting use of 3D audio. Thanks to the virtual reality and spatial sound we become part of the painting and action around it.
Binaural audio is inevitably becoming an integral part of virtual reality experiences. If you want to learn about VR or produce content you need to take audio into account. The more inputs you have the deeper you can dive into the virtual experience. Sound is equally important as the video and its role will definitely be growing.
by Michał Faliński
