Mixing music with machine learning
Imagining a new feature on behalf of Spotify* (but not actually from Spotify)…
This week, we are pleased to announce a major new feature on Spotify*.
ML Mixer is our new feature that now puts you in control of our Discover algorithm!
Your Discover playlist can now be based on how much importance you give to four key algorithm variables that are used to create your personalized playlist.
Our traditional Discover algorithm was made up of a combination of:
- the music in your library,
- the music you have been listening to lately, and
- the music that all of our other users (with similar tastes to you) are listening to.
Now we will add:
- the music your friends are listening to.
In ML Mixer, we give you these four variables as individual mixing ingredients. Each week, you can slide up and down the level of influence you would like each ingredient to have on your next playlist.
Some weeks, you may be completely bored with all of the music you have been listening to lately, so you can slide down on your own listening tastes and increase, for example, the amount of influence your friends’ music choices will have on your playlist.
Or maybe you want to listen to “classic you”, so you reduce the influence of other listeners, your own recent behavior, and your friends’ music , and you push up on the lever that represents your own music archive.
ML Mixer is based on our hugely popular Discover playlists, which provide you with a machine learning-curated weekly list of music that reflects your personal interests and tastes but which we are sure you have not heard before. Because with our data, we know everything that you have listened to on Spotify so far, but not in a creepy way.
Our Discover playlist algorithm is based on the playlists of millions of Spotify users which we compare algorithmically against what you have been listening to. We then identify songs that other users with similar tastes to you have listened to, and that you might like to hear.
(Our Discover playlist algorithm also rules out particular behaviors. For example, we don’t count a song as being part of your “taste profile” if you skip ahead on it within the first thirty seconds. Also, if you listen to whalesong at night to get to sleep, we don’t add river stream gurgling sounds as your musical preference for next week’s Discover suggestions.)
Our machine learning and data analytics team has been working hard on this feature, initially aimed at advancing some of the social aspects of Spotify. We know we need to do more with social than just allow users to follow their friends. Music is a great connector between people, and we want to leverage social in a new way that recognizes that. Already we have partnered with Tinder, for example, so that date seekers can share their listening tastes from Spotify. Our goal is to help would-be matches determine if two or three steps down the road, they are going to have to worry about cohabiting with someone who is into juggalo music.
So, at Spotify, we have been considering adding an additional algorithm variable to our Discover playlists to encourage users to discuss the music they are listening to with their peers and friends. We know there is some sharing of playlists amongst Spotify users, but, to be honest, we expected more. It’s like everyone is only ever really interested in talking about their own tastes and preferences all the time. (This is clear from some of our user research that showed that when someone asks “what music are you into?” they are basically then just waiting for you to finish talking so they can tell you what they like.)
We know music plays an important place in people’s lives as entertainment, as a form of connection, as a way to celebrate your own sense of identity, and as a social lubricant. So using our Discover algorithm as the base, our machine learning and data analytics scientists looked at the possibility of weighting more highly what people in one’s own social circle are listening to. (That is, when we compare your playlist to others, we could give extra weight to music tastes coming from the people you follow on Spotify, but also from your common social network connections on Twitter, Snapchat, Instagram, Pinterest, LinkedIn, Facebook, Tinder, and Scruff.)
For example, if, as a Spotify user, you share your online calendar with us, we could identify your planned meetings with friends in the upcoming weeks. If those friends are also users of Spotify, we could increase the influence their music tastes will have on your Discovery algorithm for the week before the meeting so, if nothing else, you have more things to talk about on your upcoming dinner or coffee catchup.
Our machine learning and data analytics scientists, however, couldn’t agree on whether you would want to pre-empt your friends’ recent music tastes or, post-meeting, discover with a sense of serendipitous wonder some of the tracks and artists your friend may have mentioned if the subject of your conversations had drifted into what-are-you-listening-to-recently territory (where, let’s face it, you were probably just waiting for them to finish talking so you could say what you have been listening to).
In the end, our machine learning and data scientist team was locked in a tied vote between whether the algorithm should factor in a friend’s listening behaviors before meeting or post-meeting, and since the team had all been working on weighting variables, at this point we started to lose precious work time because they then wanted to break the impasse by creating a new voting method where those scientists who had more friendships were given a higher weighted vote count.
We also wanted to address a challenge with using the Discover feature regularly over a long period of time. We frequently hear some comments — often from the same user, and often within a two- or three-week time span — oscillating between “you get me, you really get me” and “do you even know who I am?”
Our research shows that, with regular use over time, if you are using Discover and saving the tracks you like, then instead of widening your potential musical experience, the algorithm can inadvertently narrow its recommendations to you. For example, if you are into house music, over a couple of months of saving the house tracks that we send you, our algorithm will increasingly think you like only this type of music, until the Discover playlists we send you will all sound like bathhouse house music (i.e. umshta-umshta-umshta with an occasional high hat).
Given that impasse on social weighting from our machine learning and data analytics team, and this potential risk of narrowing your music choices rather than expanding them, we surveyed current examples of machine learning recommendation engines from online retailers, travel sites, and social media feeds to see if anyone else had solved this problem yet. By looking around at the fairly clunky recommendation engines and advertising retargeting going on, we realized that the biggest problem we all have is that we are trying to provide our users with a machine learning end-product far too early in this tech’s evolution.
The truth is, even with the large amounts of data we have access to based on your behavior on our platforms, our algorithms are only taking into account a handful of external variables.
Recommendation and advertising retargeting algorithms focus on things like what you have liked or favorited recently, what you have bought, and how long you have lingered on a particular webpage. They are unable to take into account the fact that you only liked a particular photo because your grandma who is also on Instagram shared it with you, or you only bought a particular product because you didn’t have time to research the best option, or that halfway through clicking on a website you got up and made, like, your fourth cup of coffee, even though you said three was going to be your limit from now on and you really mean it this time.
Many recommendation engines also think if you have bought a product once, you want more of the same. Which is why after booking a flight to Croatia, you get a lot more discount flight deals to Croatia. And after buying a George Foreman grill, retailers think you must be starting a collection of kitchen top fat-free grills, and will send you recommendations for more. (If you click on either of those links, you may start getting ads in your browser for flights to Croatia and for new fat-free grills. Without using an algorithm, we would recommend one of each.)
When our Discover algorithm is serving you up music that our other listeners with similar tastes to you are listening to, that is still only informed by what other Spotify users are listening to, which is not necessarily a representative sample of all music listeners with diverse tastes. Other listeners like you on our platform might not have heard of some tracks that you would all love. Our algorithm does not yet factor in looking for characteristics in songs (the same tempo, similar vocal styles, similar instruments, or similar weird-ass lyrical wordplay).
Here at Spotify (again *), we want to backtrack a little on serving you a final determination from our machine learning and algorithms. And that is why we released ML Mixer.
Algorithms and recommendation engines are a new technology. In the rush to provide machine learning as an additional feature, many startups and platform giants have released clunky models that cater to the lowest common denominator. These are now influencing our choices and exposure to novelty in regressive and mainstream ways.
We need to reposition algorithms as a new tech that works in conjunction with end users.
Our own machine learning and data analytics teams here at Spotify includes 1 Latino man, 1 Indian guy, 2 Caucasian/white women and 12 white guys. It is possible a couple of the white guys might also be gay, judging from their LinkedIn profile pics (this data is based on a quick scan on LinkedIn and my gaydar, I couldn’t find a report on our hiring rates in the quite-good-but-lacking-any-diversity-metrics Spotify HRblog). That’s not that diverse a group of people in charge of setting up the variables for your Discovery algorithm.
So the algorithms we can create at the moment probably aren’t taking into account all of the experiences and influences that truly reflect your listening interests.
ML Mixer helps overcome that by helping you become part of our data scientist team.
We hope that this new feature expands your music tastes and brings you more diverse music reflective of your preferences. But also, we hope we can demonstrate to the wider tech community how algorithms should be being released to the public right now.
We all need to make our algorithms more accountable and to invite users into the conversation more on how machine learning is influencing and directing our lives. We believe that right now, in this technology’s trajectory, it is vital to make algorithms more accessible as shareable tools rather than as pre-determined end products.
With ML Mixer, Spotify has taken that leadership step.
We now put you in charge of the machine learning.
*Please note: Not actually Spotify, they had nothing to do with this post.
But I do make monthly playlists on Spotify if you are interested. Follow me on Spotify if you want a monthly mix of pop, disco, rap, hip hop, indie, and the odd Swedish house. And follow me here for a few more articles coming out on the ups and downs of future tech, AI, machine learning, and automated processes.
Thanks to Jesus Sanz for the mockup of the ML Mixer Spotify feature (*not an actual Spotify feature). Thanks also to Bede Selleck and Eugena Ossi for reviewing early drafts.
Also, if you are interested in AI and machine learning and the impact of tech on work and society, I highly recommend Azeem Azhar’s Exponential View newsletter.
