Digging into the Spotify brain

Exposing the robot

TLDR; machine learning/AI is complex but worth digging into as the core concepts are graspable without a PHD.

Despite a diverse set of playlists (decent chunks of pop, dance, classical, hip hop and lots more) and listening behaviour, my ‘Discover Weekly’ playlist contained a repeating set of artists and an increasing slant towards minimal hip hop.

Seems I wasn’t alone:

I’m getting very tired of the way the system works. As it turns out, the algorithm now knows my taste all too well, and it’s serving me the same kind of songs over and over again.

When it launched Discover Weekly was compared to the kind of mixtape friends would make back in the cassette era, so I thought I’d put this to the test and see if I could discover (boom!) how Spotify works and what could be done:

‘ROSS’ KRAZY BIRTHDAY TAPE’ courtesy of Dave Hammond

On my 17th birthday my mate Dave made me a ‘KRAZY BIRTHDAY TAPE’ and it was ace; a custom-made cover trimmed to fit the cassette case and manually-recorded, hand-written tracklist that I grew to appreciate over time.

Dave was way cooler me (still is) and so the songs selected were cooler than most of what I listened to at the time. I remember rewinding the Jamiroquai track over and over for the first few listens, only gradually letting the whole tape play and appreciating artists like Curtis Mayfield and Freakpower who at that time were entirely new to me. Dave knew what I might like but he was also imposing a bit of his own taste onto me and in hindsight this seems like an essential role for a mixtape.

This got me thinking about how the Spotify brain works, how it constructs a mixtape and what it bases this on. How Spotify uses artificial intelligence to play the role of music-picking-friend. I thought I’d dig in and try and break down some of the mechanics behind the scenes. First I re-created ROSS’ KRAZY BIRTHDAY TAPE as a Spotify playlist, easily done aside from Curtis Mayfields ‘Hard Times’ which was missing so was replaced by his track ‘It’s Hard to Believe’.

Next I used the ‘Create Similar Playlist’ function to replicate the playlist, and did this repeatedly until I had a tenth version of ROSS’ KRAZY BIRTHDAY TAPE. I then exported these playlists (this tool is super handy) into one big doc so I could dig into it:

The number of tracks in each playlist fluctuated, but the broad make-up and body of artists was consistent, so much so that it was easy to spot the patterns:

  • 10 individual tracks appeared 5 times each, in alternating patterns.
  • Charlatans appeared 10 times with 3 songs, while the second most popular artists was Brooklyn Funk Essentials appearing 6 times (again with 2 songs). Interestingly Charlatans were treated as two separate artists (Charlatans and Charlatans UK) which I assume fooled the algorithm.
  • Over 211 songs there were 79 different artists.
  • There were 34 artists who appeared just once, 10 of whom were only present on the original mixtape.

Even with such rudimentary analysis it’s easy to see some mechanics at work. Each copy is matching a set of preference and associations, which are then mirrored on the next copy. Certain popular, higher-indexed tracks are repeating and outliers are used more sparingly. I looked into what this might be and I think it’s collaborative filtering:

The underlying assumption of the collaborative filtering approach is that if a person A has the same opinion as a person B on an issue, A is more likely to have B’s opinion on a different issue x than to have the opinion on x of a person chosen randomly

In short, the ‘Create Similar Playlist’ function was piggy-backing on the intricate decision-making Dave had made with the original tape. In the same way simple language generators can recreate Shakespeare, Spotify is using machine learning to re-create the complex relationships between the music we listen to.

To observe the ‘Discover Weekly’ brain in action I needed a clean slate; my existing Spotify profile was too full of user preferences so I created a whole new profile with just two playlists, a copy of the original KRAZY BIRTHDAY TAPE and a bigger playlist with the copies created above, otherwise I’d hit an issue common in machine learning

  • Cold start: the issue that the system cannot draw any inferences for users or items about which it has not yet gathered sufficient information.
Spotify overcoming a ‘cold start’

So Monday morning happened and I didn’t get a ‘discover weekly’ playlist, rather a set of albums ‘recommended for you’. I compared the artists concerned and found only one from the KRAZY list (and it’s bastard children).

Obviously the algorithms driving ‘discover’ are different to those driving ‘similar’. The associations are clearly being driven by a different set of relationships and parameters, and so I did some more digging and found this:

Aside from being an excellent case study in how to launch a digital service to millions of people, it contains glimpses of big machine learning ideas that drive services like Spotify, Netflix and Amazon.

Without wanting to sound naive, Spotify have models trained to determine what songs and artists are similar to one another, and combine this with user data on what he/she might like in order to construct a playlist. In reality it’s incredibly complex and I’d recommend digging into this post for more detail.

A lot of the mathematics behind this is Matrix Factorisation, best described in this answer and what I understand to be the mapping of users (me) and products (songs in the case of Spotify) against the same set of fields (hip hop, R&B etc). From ‘implicit feedback (i.e. what I listen to and put in other playlists) Spotify can create a ‘sparse matrix’ (so-called because I have not given feedback on the entire Spotify music catalogue), and use this matrix to guess what other songs I will like.

Slides 8–11 describe characteristics of Matrix Factorisation

This presentation goes into more detail specifically for Spotify and Netflix; slide 74 onwards breaks down how implicit and explicit feedback is used, and also interestingly how random negatives are used to garner the latter. It’s this explicit feedback loop that I believe is missing from the ‘Discover Weekly’ service:

To conclude, I think the ‘discover weekly’ problem is one of insularity; beyond the initial analysis of your listening preferences the Spotify brain seems too weighted to your specific behaviours and not enough to outside factors. This seems as much a design issue as a mathematical one; if the service was designed to challenge the user and suggest music that may be beyond their taste, it would echo the same thought process Dave had when he pushed me as a 17 year old to enjoy a broader spectrum of music.

As someone in service design it’s tempting to see AI and machine learning as seductive blackboxes that will enable otherwise dull services to ‘know’ and ‘understand’ us. But maybe that’s missing the point. Maybe the best purpose for the artificial brain is to occasionally stretch, provoke and to misunderstand, nudging us out of the increasingly complex echo-chambers constructed around us.

I love Matt Pearson’s take on the topic:

Much of the mechanics of human intelligence remain beyond our understanding, so simulating it digitally may now seem further beyond our capabilities that it ever did.

In Matt’s language, Spotify is artificial artificial intelligence, so rather than try and be perfect replicants of our musical tastes, find a service or an angle and design to this.

I’ll continue to poke the Spotify brain and see what else I can uncover, for if ‘they’ are right and AI will take over the world, the more all of us can do to understand the machine the better.

APPENDIX: Things I found while writing this post:

  • Multilayer perceptron: a core element of machine learning that allows for understand of non-linear data. This answer draws a parallel with learning to speak.
  • Convolutional neural networks or ConvNets: machine learning models based on our visual cortex, used to understand visual information. Not so related to the Spotify technology, but in a related field and worth knowing about.