Process: Planning a Spatial Audio Experience

Compositing audio sources

Published in

The Stories Within

4 min readMay 27, 2020

This post is a part of The Stories Within series, a project by the students of Parsons School of Design as a part of the Spatial Computing Challenge under the guidance of the R&D team at The New York Times and NYC Media Lab. The project is introduced in The Future of News Series!

What is Spatial Audio?

Spatializing audio means placing different audio sources in a 3D space such that the sounds sound different as a person “moves” through this virtual soundscape. We usually hear about stereo sounds — the sounds that are heard the same on the right and left ear. But, spatial sounds allow for the flexibility and freedom of immersion by allowing customizability.

What sounds did we use to create Spatial Audio?

After collecting and organizing the audio assets for our story, we went into production. We wanted to include different audio sources to create an immersive experience. These sources included the recorded sounds from our interviews, the recorded background sounds, the added voice-over, among others. Since we were dealing with multiple audio sources, we created a nomenclature to better define these various types of sources we were using:

Stationary Audio: The kinds of audio sources which are placed in a physical space (spatial) and that stay stationary. These sounds help build a surround sound system. In our experience, the background music and voice-overs are stationary (stereo sounds).
Non-Stationary Audio: The sounds that are moving in space. They create a Doppler effect in the soundscape and make the presence more realistic. For instance, sounds of moving cars, wind, etc.
Interactive Audio: The background and ambient sounds that are placed in a physical space (spatial) and that are interactive, i.e. they change based on the head movement.
Non-Interactive Audio: The background and ambient sounds that are placed in a physical space (spatial) and that are non-interactive, i.e. they do not change based on the head movement. These sounds are locked to the head position. In our experience, the background music and voice-overs are non-interactive.
Voice-over: These contain two types of sounds: the recorded sounds from our interviews or found-footage of the characters in our story, and the recorded sounds by a professional or text-to-speech generator that are added afterward to help build the narrative.
Background Music: The ambient sounds that are present in the background.

How did we plan a Spatial experience?

To produce an audio experience, we needed a platform for assembling all our sounds so that we can hear them in unison, and rearrange them as needed. For that purpose, we used audio-mixing software (Adobe Audition).

Note: We used Adobe Audition, but any audio-mixing software would work. Sound designers prefer Audacity.

As represented in the image, we assembled all the sounds in Audition — voice-overs, stationary sounds, background music. The progressive interface allowed us to visualize the experience from start to finish. Inside Audition, we trimmed the clips and adjusted the audio levels to match that of the overall experience. Once we had a setup we were satisfied with, we exported individual clips from our timeline to go towards the next step: spatialize them.

How did we produce a Spatial experience?

After finalizing the flow of the experience and downloading individual clips, we used the Unity game engine to develop the interaction. We wanted to create a simple drag-and-drop system, to make it easy for media organizations to implement and use.

The development of this platform laid the foundation of the three types of audio experiences that we developed: Spatial Podcast, WebVR, and BoseAR. Details on that covered in the coming chapters.

About the Team

Akshansh Chaudhary is an XR Experience Designer. In his projects, he focuses on social and world issues like privacy and local news. His design approach is to create immersive and playful experiences to spread awareness. Follow his work on akshansh.net.

Karen Asmar is a design technologist working at the intersection of the built environment, society and human-computer interaction. Her work focuses on exploring the impact of emerging technologies on ways we interact in space, with space and with data in space. Follow her work on karenasmar.com.

Ponvishal Chidambaranathan is a fervent immersive storyteller and digital producer with a strong inclination towards innovative, philosophically charged content in unconventional storytelling and interactive media. Follow his work on ponvishal.com.

Yashwanth Iragattapu is a creative technologist and interaction designer. He creates products that encompasses human and spatial interactions through emerging technologies like Augmented reality and Virtual reality. Follow his work on yashwanthiragattapu.com.

Debra Anderson is an entrepreneur and educator specialized in XR. She is recognized for designing data-driven and research based approaches to immersive experiences with a focus on how data and emerging technologies can be used for civic engagement and social impact.