New Waves of Engagement: How AI Music Can Increase Live-Streaming Value
Last year’s World Championship was the first I enjoyed in ages. I didn’t watch the football competition in Russia, though. Instead, I was fascinated by the heroics of teams like Invictus and G2 Esports, casting spells and summoning magic creatures as they battled to outsmart each other on the League of Legends’ playfield. As a chess buff, I was immediately struck by the brilliance and strategic depth of this third-person multiplayer online game. And I wasn’t the only one. A combined two hundred million viewers watched the drama of the League of Legends’ 2018 Grand Final unfold, on live-streaming platforms like Twitch and Youtube Games.
These numbers demonstrate how pervasive live-streaming has become. This is especially true for gaming platforms like Twitch, where millions tune in daily to follow the incredible stunts of Ninja and other top players in games like Fortnite and Apex Legends.
I engage, therefore I am
Why would millions of users watch other people playing video games? Unsurprisingly, they want to be entertained. Users look for engaging experiences they can feel part of. When Ninja goes on a killing spree in Fortnite, the comment section explodes with messages of delighted fans fueling their idol. In these moments, followers feel they are almost one with their streamer hero.
Like Hollywood, Twitch and similar live-streaming platforms sell memorable experiences. The value chain of these businesses is quite simple. The more engaged users are, the more they’ll stick around, and the more revenue for the platform. For these services, engagement is the main currency.
Obviously, increasing that engagement is paramount for live streaming companies. But how can these platforms become more engaging? Engagement is multifactorial. Visuals, audio and interaction combine to create a memorable experience. Talking directly to Ninja through the comment section keeps fans glued to their screens.
Many companies are betting on improving the fan-streamer connection as a way to monetise Twitch users. The value proposition of StreamLoots is increased user engagement through better interaction between streamers and fans. If you’d like to have a say in choosing Ninja’s next Fortnite character, try StreamLoots. Fans can buy virtual cards during the Twitch stream which are used for voting on which clothes or character the streamer should use next round, for example.
Engaging with the music of AI
Even though improving visuals and interaction can enhance users’ engagement on live-streaming platforms, augmenting the sound experience is a more promising route. That’s because, unlike other areas, music’s potential has yet to be tapped. This is actually an overstatement. To be frank, most of these platforms have no music solution whatsoever.
Simply put, live-streaming services have traditionally faced two major music hurdles: copyright and context. Unfortunately, you can’t stream the latest Elettra Lamborghini masterpiece while wooing your die-hard fans with another game of League of Legends. That’s not just because this would be a potentially career-destroying musical crime against your followers (…and humanity?), but rather because Twitch doesn’t own the rights to stream her music. (Check out Trevor Rawbone’s piece on the issues of music copyright in game streaming for an in-depth discussion).
Even if, while streaming, Spotify’s whole library was at your fingertips, this still wouldn’t raise your engagement threshold extensively. Experiments suggest that users feel more involved when the music reinforces the game’s emotional context. How can this happen with Elettra cheerfully pem-pemming in the background while you’re dying under the fierce attacks of the honourable Baron Nashor?
Both music copyright and context problems can be addressed with the magic of Artificial Intelligence (AI). Let’s explore three examples of how AI can increase the audio experience, therefore increasing engagement, therefore increasing the time users spend on the platform, therefore… you know where I’m going with this.
Recommending royalty-free music
The Internet is filled with royalty-free music libraries. Ask indie game developers, and many will quickly praise the quality of some of the music in these catalogues. Imagine a live-streaming platform strikes a deal with one of these libraries, granting their users free use of the entire catalogue. This would be a fraction of what’s usually paid to merely sit at the table with the music label big boys in the hopes of negotiating a licensing deal.
Here’s how AI fits into the puzzle. The streaming platform could build a music profile that captures each streamer’s musical taste and preferences. How would a platform do that? By analysing streamer’s Spotify accounts, or, even simpler, examining the musical choices of the users directly on the streaming platform.
Let’s assume the platform discovers that Ninja can’t live without uptempo K-pop (wouldn’t you know it?). He decides to create a new playlist for his Fortnite streams. An AI recommendation system can propose a selection of uptempo songs from the royalty-free library. These pieces sound similar to what Ninja usually listens to on Spotify. (For a primer on music recommendation systems, you can have a look at an article I wrote about Spotify’s Discover Weekly.)
With a built-in recommendation system, a platform like Twitch could avoid copyright issues. The system also saves streamers time when creating music playlists. In other words, music acquisition gets streamlined by serving stylistically relevant music to the streamer.
Mood-based infinite music stream for streams
As you’ve probably figured out, the major issue with the music recommendation system is that it doesn’t know anything about the context of the stream. When Ninja’s on a rampage, the music should be excited. To achieve a strong connection between music and stream context, there should be a way for the music recommendation system to make intelligent decisions based on what’s happening in the stream. AI, once again, provides the solution.
What we’re trying to achieve is an infinite radio-like application that streams context-relevant music, and is based on the streamer’s musical taste. In plain English, when Ninja’s character suffers a deathblow in a Fortnite standoff, we may want some angry Norwegian death metal to kick in that coordinates perfectly with all the unspeakable curses Ninja is shouting at the screen.
This application is a fancier version of the music recommendation system discussed previously. We can already count on a music profile of the streamer. But we’re missing a way to gauge the emotional context of the stream. For this, a composite AI model could be trained to use at runtime during the streaming sessions. This would analyse the audio coming from the game, the streamer’s commentary, and the comments left by the users; in order to understand the sentiment and the events unfolding. Once it has contextual data, the system can spontaneously serve up music with a mood and style that matches the stream’s emotional state.
The emotional link between the music and the stream increases users’ engagement. However, there’s a drawback. This system presents a significant lag in responding to a changing streaming context. The average song is three minutes long. Once the system serves up a new song, it has to wait three minutes before playing the next one. In the meantime, the emotional context can obviously change. For example, the system can play Ponponpon (a Ninja favourite) while Ninja cheerfully picks his Fortnite character. Two minutes into the game, he may face a life-or-death situation. The jolly Ponponpon has now become an unfitting sadistic ode. Unless we decide to fade out the song, a painful wait would ensue before the music changes. To solve this issue, a live-streaming platform could embrace a more sophisticated AI music solution — realtime music generation.
Realtime music generation
Several companies have already developed technology that leverages AI to generate music from scratch, notably Melodrive, Magenta (Google) and Jukedeck. A few of these systems can directly create realtime music in-game. It’s like having uber-fast John Williams packaged in a game. Initially, he waits to see what the player does. Then, he scribbles a score in a microsecond, then, the microsecond after, moves to the podium and conducts the score with an orchestra. Finally, the microsecond after, he’s in the studio to mix and master the audio.
Imagine the opportunity provided by integrating this pseudo-John-Williams-technology in live-streaming platforms. Ninja starts a Fortnite game. Everything is calm as he gently parachutes onto a grassy hill. The AI reads the stream’s emotional context and provides relaxed downtempo music. The first enemies slowly appear on Ninja’s radar. The AI identifies these cues and responds, increasing the pace of the music, throwing in a few dissonances here and there. Damn! Headshot by a sneaky sniper (Where were they?!). The generated music ramps up its tension following the sudden emotional burst from the gameplay. You get the idea.
With realtime music generation, the music adapts instantly to the game context and changes its composition, performance and synthesis elements accordingly. The fact that the music aligns perfectly with the emotional context of the streaming, increases the users’ engagement, and, ultimately, makes their experience more memorable.
A final stream of consciousness
Live-streaming platforms are redefining entertainment. Young people from all over the world tune in to platforms like Twitch to see their gaming idols perform virtuoso stunts. If you’re part of the old guard that can’t really understand this new phenomenon, just pause and consider why many of your friends eagerly anticipate Sunday afternoons to watch 22 people try kick a ball into a net attached to a set of poles (or any other sport, really). Your friends, and the Twitch users, are probably driven by the same basic instinct: We all want to be entertained and live memorable experiences, whether they’re on a football pitch or in the League of Legends arena.
Live-streaming platforms have developed numerous ways to increase users’ engagement, mainly focusing on visuality and interactivity. For logistical reasons, the potential of music as an engagement multiplier has been long overlooked. With the help of AI, these platforms can score easy goals by significantly improving their user’s music experience. This, in turn, will increase users’ engagement; the fundamental ingredient to stage memorable experiences.
Before you go, remember to sign up to The Biz of AI mailing list. Once you sign up, you’ll start receiving weekly emails with content that you can read at your own pace.
