WebVR: Developing a Spatial Audio Youtube 360 Experience

Development of interactive spatial audio experience

This post is a part of The Stories Within series, a project by the students of Parsons School of Design as a part of the Spatial Computing Challenge under the guidance of the R&D team at The New York Times and NYC Media Lab. The project is introduced in The Future of News Series!

Due to COVID-19, people had to stay at home and avoid going outdoors. This situation has brought new challenges in terms of accessing local stories at the location. In order to give a location context to the spatial stories, we developed a WebVR experience that can be accessible to anyone who owns a smartphone.

To create this spatial audio WebVR experience, we need to use a platform that supports audio spatialization. There are many tools/software out there that can spatialize audio and give a 360 audio experience to the users. We specifically used the Unity game engine because it is a 3D environment that allows users to easily visualize audio components arranged in the space around them. Also, Unity is a commonly used platform for developing VR experiences.

The Github link to the project is provided here. Create your own spatial audio VR stories easily by just adding the audio clips and setting some parameters which will be explained below.

Link to GitHub Page!

Recommended Unity 2019.3 and above

• Once the project is loaded open Edit -> Project Settings and in Audio, make sure Spatializer Plugin and Ambisonic Decoder Plugin is set to Resonance Audio.

• In Project tab open Scenes -> Spatial Audio VR Experience
• In the Hierarchy, [Audio Clips] Audio Components contain all the GameObjects that hold various types of audio sources. There are 4 types of audio sources, Background Music, Voice-Over, Interactive Stationary sounds and Interactive Non-Stationary sounds. We will be using these GameObjects to add our own audio clips and create a complete experience.

• To add spatial stationary sounds in the experience, select Interactive Stationary Sounds GameObject and in the inspector to the right specify the number of audio clips and set various parameters of these audio clips.
  Clip — Attach audio clips that need to be spatialized
  Start Time — The time (in seconds) at which this sound needs to play
  Duration — How long the clip should be playing
  Volume — The volume at which the clip needs to play
  Index — Location where the sound needs to play
• The convention for the index is as follows:
  0 — North (N)
  1 — North East (NE)
  2 — East (E)
  3 — South East (SE)
  4 — South (S)
  5 — South West (SW)
  6 — West (W)
  7 — North West (NW)
  8 — Up

• To add spatial moving sounds in the experience, select Non-Interactive Stationary Sounds GameObject and in the inspector to the right specify the number of audio clips and set various parameters of these audio clips.

• After adding all the sound clips and setting the parameters, the sound experience is now done.
• The next step is to add a 360 skybox to bring the location context. In order to do that we use a 360 image to create a skybox material. Steps to creating a 360 image skybox material.
• The final step is to build the project on Android or iOS in order to get VR experience. We use the GoogleVR package to create VR experiences.
• To make a build for Android,
  - Open File -> Build Settings and select Add Open Scenes
  - Select Android and click on Switch Platform
  - In Player settings, change the company name and the product name
  - Under the android tab in XR Settings, check Virtual Reality Supported and add Cardboard by pressing ‘+’ on the bottom right
  - In Other settings, remove Vulkan from Graphics API by pressing ‘-’
  - Set the Minimum API level to Android 5.0 ‘Lollipop’ (API level 21)
  - Make sure Package Name is ‘com.companyname.productname’
  - Press build and name the apk or if android phone is connected, select the device and press build and run

There is a way to create VR experiences that can exist on the web. We can record a 360 video on Unity using Unity recorder and then create a Youtube 360 experience by following the tutorial How to Edit 360/VR Video in Premiere Pro CC 2017 (Monoscopic & Stereoscopic)

The following are the resources used in this project:

1. Resonance audio SDK by Google — Resonance audio is a spatial audio SDK that enables developers to create more realistic VR and AR experiences on mobile and desktop
   Link to get started in Unity — https://resonance-audio.github.io/resonance-audio/develop/unity/getting-started.html
2. Google VR — This package is used to build apps for usage in Google Cardboard or Daydream
   https://developers.google.com/vr/develop/unity/get-started-android
3. iStreetView — To download 360 images from street view on google https://svd360.istreetview.com/
4. Creating Youtube 360 experience using Adobe Premiere Pro https://www.youtube.com/watch?v=-xNeooQ8tAE&feature=youtu.be

About the Team

Akshansh Chaudhary is an XR Experience Designer. In his projects, he focuses on social and world issues like privacy and local news. His design approach is to create immersive and playful experiences to spread awareness. Follow his work on akshansh.net.

Karen Asmar is a design technologist working at the intersection of the built environment, society and human-computer interaction. Her work focuses on exploring the impact of emerging technologies on ways we interact in space, with space and with data in space. Follow her work on karenasmar.com.

Ponvishal Chidambaranathan is a fervent immersive storyteller and digital producer with a strong inclination towards innovative, philosophically charged content in unconventional storytelling and interactive media. Follow his work on ponvishal.com.

Yashwanth Iragattapu is a creative technologist and interaction designer. He creates products that encompasses human and spatial interactions through emerging technologies like Augmented reality and Virtual reality. Follow his work on yashwanthiragattapu.com.

Debra Anderson is an entrepreneur and educator specialized in XR. She is recognized for designing data-driven and research based approaches to immersive experiences with a focus on how data and emerging technologies can be used for civic engagement and social impact.

The Series List:

• Introduction: The Future of News Series
• Process: Exploring the Process of Immersive Audio Stories
• Process: Planning a Spatial Audio Experience
• Spatial Podcast: Developing a Spatial Audio Experience
• WebVR: Developing a Spatial Audio Youtube 360 Experience
• WebVR: Creating a 360 Image Background for VR
• BoseAR: Developing a Spatial BoseAR Experience