A Case Study on Designing Sound for Immersive Education
A look into how I used sound for information, interactivity, and immersion within NC State DELTA .
I’ll go ahead and argue that interactivity and sound are going to distinguish a VR application from any ordinary immersive experience. That’s the conclusion I made after working on the NC State DELTA team on exploring 360 video platforms for an event planning class. We took an exploratory journey on how to create content for VR goggles (namely the Oculus Go) using a paid authoring platform called CenarioVR (Think Squarespace for VR). Though I could do a separate blog on how we got to choosing those technologies, I mostly want to focus on the strategies for using designing sound for these commercial platforms. Sound is often secondary for design, so I’d to shed some light on how to give sound a better presence.
While I would love to explain how everything works in a two-thousand-word essay, I think a video walk-through of part of the final product might illustrate what exactly I’m talking about (for those unfamiliar with 360 video). The embed below is a screen capture of myself going through one of the authored experiences on the Oculus Go
I am going to categorize sound design for VR into four main categories: Voice-over, sonic interaction design, spatial sound design, and accessibility. I posit that all of these elements are necessary for an engaging and inclusive experience for students.
Voice-over
Recording the instructor’s voice was a great way to deliver guidance to the viewer, as text in VR is really difficult to get right. More often than not in education, we have to communicate a lot of content over a short period of time, and voice-over is a fantastic way let the student explore a visual experience while still being guided by the instructor. We’re working on a way to include captions for the hearing-impaired, but most platforms (including CenarioVR) do not support them yet.
I’ll be the first to admit I’m not an expert on sound production, but there are a plethora of wonderful YouTube channels that can help you start out. I relied heavily on the advice of Booth Junkie to make the voice-over cleaner and easier to understand. After that, it was a pretty straightforward workflow until you start compiling things together.
Our biggest problem was condensing the length of the voice-over to be an average of 30–45 seconds. Longer than that, and you start to lose the interest of the student because the video has already ended. You have to remember that VR is not a replacement for the textbook — it’s more of a compliment and an enrichment to long form writing and reading.
The other issue was when the voice-over and recorded audio from the video conflicted. If the student plays the voice-over while people in the video were speaking, the student won’t be able to hear either. As a part of good user experience, we programmed into CenarioVR the inability to play both the video and the voice-over at the same time if there’s a conversation in the video.
Interaction Design
The 360 platform we used, CenarioVR, afforded us the ability to easily program interaction with graphics and sound.
Graphics were superimposed onto each scene. But I wanted to join in on the fun, so I used what’s called an earcon to add depth and interest to the each of the scene links. An earcon is like an icon, but uses sound to represent or symbolize an object. The video below is demonstration of how earcons work. Whenever the user hovers over an icon, the user can ‘hear’ that icon as confirmation. You’d be surprised with the physical responses (laughter, attention, and playfulness) we got with them in our user-testing.
I used Audacity almost exclusively for this part of the design. It’s a platform I’m most familiar with and made it really easy and simple to prototype new sounds without messing with fancy plugins.
My favorite that I came up with was the ‘Replay’ earcon. Using a tape deck as a metaphor to aurally describe the process of starting a scene, I used recorded samples of a rewind, play, and stop.
Spatial Sound
CenarioVR had strong limitations when it came to spatial sound. You could upload mono audio files into the scene and place the sound in the experience (which I’ll admit is a great starting point for beginners), but there isn’t an option to upload a video with multichannel audio.
It was quite clear that CenarioVR was assuming a lot about their users might want to use spatial audio. A good 360 watching platform, like Showtime VR, will include support for both First-Order Ambisonics (AmbiX Format) and Facebook 360 Format for soundfield reproduction. Our solution, with the remaining time that we had, was to just use mono audio for the entire experience. Without the fine-grain control of the soundfield that I wanted, I didn’t think it was worth getting into.
But wait! Why do you have a whole section on spatial sound if you didn’t even use it for this project? There’s a reason for that:
You should not use stereo audio with 360 video.
If there’s anything you should take away from this section it’s that stereo audio does not work well with 360 video unless you know how to do head-locked stereo on top of ambisonics. Unfortunately there are those who upload 360 videos with just regular stereo (even though they know better), without thinking about what happens when you turn your head. I cringe every time. Alright, I’m done with my spiel, let’s move on.
Accessibility
There are a few accessibility concerns that we addressed, and others that weren’t addressed with this project. Those will be included in the next project. We’ve so far done testing to make sure that everything you need to interact is in front of you, so that people with mobility challenges don’t need to look behind. And I’m certain that the graphics were tested for red-green colorblindness.
Having a desktop version (like CenarioVR does) is a great way to include people who get motion sickness. Most platforms also have a deployable version that you can view as a web or desktop application. It’s not VR per se, especially with mono audio, but it’s still like playing a 3D game.
In future projects, when we get to use our own custom Unity platform, we’ll have the ability to implement accessible options for the 360 viewer. Our plans might be as follows:
— A magnifying glass to see text better
— Captions
—Contrast and Brightness adjustment for people with low vision
— Audio-only version for the blind
As the field develops it will be more and more important for the technology we create to be accessible so that it’s a process that everyone can be a part of. Especially when it comes to education, of all fields, accessibility and user experience are essential components to design.
Here’s my favorite video on accessibility in VR.
A Note to VR Designers: Audio should have equal footing!
To wrap things up, I want to add that sound is not often a primary consideration for design for VR technology, even though it provides about half the experience. It gets easier to do sound for VR with free or affordable tools like Reaper (affordable), The Ambisonic Toolkit (free), Audacity (free), and the Zoom H3VR (affordable).
I would also recommend the Immersive Audio Podcast for enthusiasts. It’s look into where the technology is and where it’s going in the next decade.
If you’re making educational content, I think the more opportunity to verify with potential users and instructors that what you’re making works, the better. User-centered design is a methodology that I highly subscribe to and hope to get better at.
If you have any questions, comments, etc., feel free to reach out at jtkuhn3@ncsu.edu
