30% More Immersion in VR — An Adaptive Music Experiment
The following is an account of an experiment Melodrive conducted about a year ago. We wanted to understand more about how adaptive music affects immersion in virtual reality. The results were amazing — not only did they exceed our expectations, but they altered our understanding of the music experience altogether. Since we celebrated our second birthday yesterday, we thought it would be fun to re-share this research that guides everything we do.
This article originally appeared on the Melodrive blog, 19th February 2018. It has been copy edited for clarity.
There’s no doubt that VR is a highly immersive medium. You don’t have to spend a week wearing a VR headset to figure this out (although this guy did just that). VR players don’t just see the world through a screen, as is the case with traditional video games; they’re literally inside that world, interacting with its objects and environments intuitively. Most of my friends who tried VR for the first time where shocked by the experience. They told me that they completely lost track of time and felt as if they were transported to an alternate reality. Simply put, they were deeply immersed.
There are several factors that enhance the immersion of an experience. One that’s undoubtedly important, yet often overlooked and underappreciated, is music. Composers have always thought that music has the ability to increase the level of immersion of players experiencing digital content, whether in videos, video games or VR. For interactive content, composers like Guy Whitmore, who are at the forefront of music-making in non-linear settings, know that adaptive music can be the difference-maker in terms of immersion (check this post for an explanation of what we mean when we say adaptive music). The reasoning is fairly simple. With adaptive music, no matter how the user behaves, the music is always in sync with the emotions portrayed in the visuals and the storyline. Here’s an example. My jolly village gets attacked by an army of dark knights. The music, being adaptive, dynamically shifts from happy to dramatic. The double bass kicks in and the chords get more aggressive. In other words, the audio elements of the experience reinforce the story told through the visuals. Composers suggest that this reinforced feedback between different elements of an interactive experience increases immersion. This is intuitive and sounds like a plausible hypothesis, but it’s still a hypothesis. No one had tested it yet in the real world — until now.
The Experiment
So far, correlation between immersion and adaptive music has never been tested scientifically. Our team was curious enough to run our own psychological experiment to shed some light on this. In particular, the goal of our study was to understand the impact of music on immersion and on the time spent in a VR experience. Our main hypotheses, based on the intuition of composers and ourselves, were the following:
- The presence of music increases the level of perceived immersion in a VR experience.
- Deep adaptive music increases the level of immersion more than linear music.
- The presence of music increases the time spent in a VR experience.
- Deep adaptive music increases the time session in VR more than linear music.
- Deep adaptive music fits a VR scene better than linear music.
First, I need to clarify some concepts so that you grasp the full picture behind these hypotheses. You may wonder what deep adaptive music is, compared to linear music. Linear music is what you’re familiar with; it’s what we listen to all the time, for example on Spotify, and it never changes no matter how much you’d like it to. It’s a recorded composition which you can loop indefinitely, and always contains the exact same beginning, middle section and ending. By contrast, deep adaptive music is dynamic — it’s generated in realtime by an Artificial Intelligence that adapts to the emotional setting of a scene and the user’s interaction, on the fly. Deep adaptive music is infinite in the sense that it will always differ slightly for each player, depending on their interactive experience. That’s because each player will explore a VR environment in a unique way.
Methods
To test our hypotheses we built a simple VR scene. The scene consisted of a space station with two rooms connected by a corridor. The first room (a blue room) was peaceful and calm, the second (a red room) was more aggressive. We modulated the mood of the rooms (ranging from tender to angry) by using different lighting and levels of activity in the same objects present in both rooms.
We set three experimental conditions. Participants could either explore the VR scene without music, with linear music or with deep adaptive music. The music used was in an ambient-like style with electronic instruments. Have a look at the table below for a comparison between the three conditions.
A total of 46 participants took part in the experiment. We didn’t tell them what the experiment was testing, so they weren’t biased. Participants experienced only one of the three conditions and were instructed to explore the scene for as long as they liked. We tracked session time as an overall measure of engagement. Once participants were done exploring the VR experience, they were asked to fill out a questionnaire on music and immersion.
Results
The day after we conducted the experiment we started analysing the data we collected — and we were amazed by the results. The initial hypotheses we came up with were found to be true and supported by the empirical results. (For the statistics nerds among you, all the results I discuss next are statistically significant with p<0.05).
The effect of deep adaptive music on immersion was large. We found that deep adaptive music boosted the sense of immersion by 30% when compared to no music, and 25% when compared to linear music.
The effect of deep adaptive music on time session was even more impressive. With deep adaptive music there’s a 42% boost in time spent in the VR scene over no music, and a 27% increase over linear music.
90% of people agreed that music was an essential component in helping them feel immersed. The data shows that deep adaptive music contributes significantly more than linear music to the sense of immersion.
We also found that deep adaptive music increases the match between music and the VR scene by 49%, when compared against linear music.
Qualitative Feedback
The qualitative feedback we got from participants was also revealing. A few participants who explored the scene with no music were somewhat bored and thought that music could make the experience more immersive:
“The silence made the entire experience very bland… the lack of sound took me out of the immersion.”
“It was interesting, would have been better with music.”
Participants thought that, overall, the music was good and fitted the visuals well. However, some of those hearing linear music thought it wasn’t changing to fit the unfolding experience:
“I was listening closely for subtle changes in the music to indicate progression in some way, but it felt as though my actions and progressions through the rooms had no impact. I still enjoyed the music and it fit the theme very well.”
“I really enjoyed the atmosphere of the environment and the music up until the red room. At that point, it felt like the tone of the music no longer matched that of my surroundings.”
Participants who got deep adaptive music noted that the music changed according to the changes in the scene. This reinforced their emotional journey:
“I liked the sound. It was building tension. As I was moving through the scene I could feel it became more intense. When moving away from the red room the tensions decreased again.”
“The music did a great job of blending transformations based on the scene, adding arrangement layers when less or more action was present.”
What Have We Learned?
The psychological experiment we conducted was conclusive. Finally there was empirical proof for the long-standing assumption that music significantly increases immersion and engagement in interactive media. And we now know that dynamic music fundamentally increases time spent in a VR experience. In light of the findings of this study, we can easily say that dynamic music can increase immersion by one third and almost increase the time session by half. That’s an incredible impact.
For me, the most striking element that arises from the experiment is not the huge impact of music as a whole. Being a composer and a gamer, I was already convinced that music, being an effective medium to convey subtle emotional cues, would have a major impact on engagement. What I’m really shocked about is that there’s a huge difference between the impact of linear and deep adaptive music in VR. Of course, I was expecting for deep adaptive music to have a stronger effect on immersion and time session. But I didn’t imagine the difference could be so strong. The results suggest that deep adaptive music is what really makes the difference for engagement.
This is also evident from the qualitative feedback that we collected. Participants were particularly good at spotting mismatches between visuals and music, and appreciated and enjoyed when the music coordinated with the VR scene. This is the final take-away for me. Music is key for engagement in VR, but to be more specific, deep adaptive music is what can make the difference in such a highly non-linear medium.
Think of how overlooked music is in VR right now and, nonetheless, how immersive of an experience VR is. Now, imagine you add the right emotion-evoking music that’s also deeply adaptive. In one simple step the least you’ll achieve is a one-third increase in immersion. That’s worth exploring.
The preliminary results of this study have been published at an international workshop. Check out the conference proceedings (our paper is on page 8) if you’d like more details about the experiment.
