Spatial Podcast: Developing a Spatial Audio Experience
Development of a head-locked spatial audio experience
This post is a part of The Stories Within series, a project by the students of Parsons School of Design as a part of the Spatial Computing Challenge under the guidance of the R&D team at The New York Times and NYC Media Lab. The project is introduced in The Future of News Series!
To create a spatial audio experience, we need to use a platform that supports audio spatialization. There are many tools/software out there that can spatialize audio and give a 360 audio experience to the users. We specifically used the Unity game engine because it is a 3D environment that allows users to easily visualize audio components arranged in the space around them.
The Github link to the project is provided here. Create your own spatial audio stories easily by just adding the audio clips and setting some parameters which we will be explaining below.
Recommended Unity 2019.3 and above
- Once the project is loaded open Edit -> Project Settings and in Audio, make sure Spatializer Plugin and Ambisonic Decoder Plugin is set to Resonance Audio.
- In Project tab open Scenes -> Spatial Podcast
- In the Hierarchy, [Audio Clips] Audio Components contain all the GameObjects that hold various types of audio sources. There are 4 types of audio sources, Background Music, voice-over, Interactive Stationary sounds and Interactive Non-stationary sounds. We will be using these GameObjects to add our own audio clips and create a complete experience.
- To add spatial stationary sounds in the experience, select Interactive Stationary Sounds GameObject and in the inspector to the right specify the number of audio clips and set various parameters of these audio clips.
Clip — Attach audio clips that need to be spatialized
Start Time — The time (in seconds) at which this sound needs to play
Duration — How long the clip should be playing
Volume — The volume at which the clip needs to play
Index — Location where the sound needs to play - The convention for the index is as follows:
0 — North (N)
1 — North East (NE)
2 — East (E)
3 — South East (SE)
4 — South (S)
5 — South West (SW)
6 — West (W)
7 — North West (NW)
8 — Up
- To add spatial moving sounds in the experience, select Non-Interactive Stationary Sounds GameObject and in the inspector to the right specify the number of audio clips and set various parameters of these audio clips.
- After adding all the sound clips and setting the parameters, the complete experience is now done. The next step is to export audio. To do that we use the Unity Recorder package from the package manager. This package is pre-installed in the project.
- Select Window -> General -> Recorder -> Recorder Window
- Select + Add New Recorders and add Audio
- You can change the destination of the folder and name of the file and then click on “Start Recording”
- The recording that is obtained is the spatial podcast and it is ambisonic format so no extra processing is needed.
The following are the resources used in this project:
- Resonance audio SDK by Google — Resonance audio is a spatial audio SDK that enables developers to create more realistic VR and AR experiences on mobile and desktop.
Link to get started in Unity — https://resonance-audio.github.io/resonance-audio/develop/unity/getting-started.html - Unity recorder — https://docs.unity3d.com/Manual/com.unity.recorder.html
About the Team
Akshansh Chaudhary is an XR Experience Designer. In his projects, he focuses on social and world issues like privacy and local news. His design approach is to create immersive and playful experiences to spread awareness. Follow his work on akshansh.net.
Karen Asmar is a design technologist working at the intersection of the built environment, society and human-computer interaction. Her work focuses on exploring the impact of emerging technologies on ways we interact in space, with space and with data in space. Follow her work on karenasmar.com.
Ponvishal Chidambaranathan is a fervent immersive storyteller and digital producer with a strong inclination towards innovative, philosophically charged content in unconventional storytelling and interactive media. Follow his work on ponvishal.com.
Yashwanth Iragattapu is a creative technologist and interaction designer. He creates products that encompasses human and spatial interactions through emerging technologies like Augmented reality and Virtual reality. Follow his work on yashwanthiragattapu.com.
Debra Anderson is an entrepreneur and educator specialized in XR. She is recognized for designing data-driven and research based approaches to immersive experiences with a focus on how data and emerging technologies can be used for civic engagement and social impact.
The Series List:
- Introduction: The Future of News Series
- Process: Exploring the Process of Immersive Audio Stories
- Process: Planning a Spatial Audio Experience
- Spatial Podcast: Developing a Spatial Audio Experience
- WebVR: Developing a Spatial Audio Youtube 360 Experience
- WebVR: Creating a 360 Image Background for VR
- BoseAR: Developing a Spatial BoseAR Experience
