Mixed reality has presented to us new ways to teach skills and solve problems previously impossible. Whether onboarding new employees or assisting first responders remotely, the use of VR and AR allows developers and creators to assess situations in innovative ways. In the case of training, immersive experiences have proven to have high rates of learning retention, and, in the field, AR can be a useful communication tool between those on the ground and in a command center.
The Importance of Psychoacoustics
No matter how good motion controls or graphics within VR and AR get, true immersion cannot be achieved without paying attention to psychoacoustics. Psychoacoustics essentially defines how humans perceive sound, and the tenants of this study can be utilized in VR to maintain the masquerade of reality. Some people who may have never experienced mix reality before may be put off by the unnatural aspect of the experience. Life-life, object-based audio is key to helping users acclimate themselves to an XR environment. Audio acts as the door into interactivity, and the ability to localize directional audio helps users feel as if they have control over their environment. The longer an illusion of realism is maintained, thanks in part to object-based auditory cues, the longer a user remains in the headset so long as their immersion is uninterrupted.
XR has become invaluable in training, especially when simulating situations that would be unsafe for people to be in or impossible to recreate in real life. In the Dima Lab project, VR is used to train caregivers for patients with dementia. The Dima Lab experience allows trainees to step into the shoes of their patients by projecting unto the interacting party hallucinatory episodes that a dementia patient may experience. That caregivers are given the opportunity to experience what their patients are seeing from a first-person point of view is something that would be impossible without VR and allows caregivers to empathize with their patients in new and informed ways.
In designing audio for this project, it was important to keep in mind the point of view of the user. Those who are in this experience are embodying a human with a medical condition rather than an omnipotent viewer, and it was important that the audio reflected that. So auditory cues were head locked and condition-based audio interactivity to accommodate hand tracking was implemented, allowing for an experience that one caregiver says changed the way he interacts with his patients.
In the Field
Search and Rescue 5G (SAR5G) is an AR application that utilizes 5G to support emergency relief teams on the ground and remotely increase communications and improve decision-making strategies with remote resources. In high-stress environments where visibility may be inhibited, high-quality audio is essential to assisting a team with navigation and awareness. Auditory cues within a user interface can be used to direct the wearer to a certain point, or notify them of any changes or dangers in their immediate environment.
There are many factors to take into consideration when designing audio for an application like SAR5G. Since the application is intended to be used on the ground, the noise level of the surrounding environment and possibility of visual impairment were a couple challenges that needed to be designed for. It is important to keep the application’s audio at a level above the environment’s noise floor. In action, the noise floor could be measured by onboard microphones and adjusted for on the fly with presets built into the application that adjust the mix as necessary. This is especially important levels of visibility are low as auditory cues become critical to locating fellow scouts, escape rooms, or victims, to name a few.
As mixed reality continues to grow, more uses for the medium will be discovered and it will become ever more essential to our daily lives. Without attention being paid to the auditory design of these applications, their effectiveness is lessened, whether it be in the level of immersion that can be achieved or the basic functionality of the program. With spatialized audio, XR can be taken to the next level.
For more information on the application of psychoacoustics in XR, feel free to reach out and look forward to a video of our talk at GameSoundCon.