Nov 9, 2021
#22. Spotify: A Product Story ‘Review’
04: Human vs Machine.
Dear Esteemed Reader,
I hope this finds you well. In our last two articles we reviewed 2 incredible books on building digital products and interviewing potential users for vital insights: Inspired and The Mom Test.
Today we return to our exciting series on Spotify — Spotify: A Product Story.
Besides learning about how to build a great product, transform your industry, and grow your business to 300m users, you would also learn about making strategic business decisions.
Summary
Today we shall learn about how Netflix, Amazon, and Spotify build recommendation systems, applying machine learning to data, and how to build ‘machine learning first’ products.
These are the 4 key lessons we would learn:
Lesson #1: Build for yourself first. But don’t build for yourself only.
Lesson #2 — Throwing Machine Learning at data you don’t fully understand isn’t enough to give you a great product. You need to understand your data deeply.
Lesson #3 — If you don’t have one side of an equation inside the company — look outside for it. And weigh the present-day costs against how far ahead you can leapfrog into the future.
Lesson #4 — In a machine learning world, you have to learn the product, not build it!
QUICK ANNOUNCEMENT
I would be taking a 1-month break from this newsletter to focus on completing my dissertation and my program.
The title of my dissertation is: The Impact of VR on Customer’s Perception of African Tourist Destination. I am fascinated about how my research can support the post covid recovery of this key industry.
To support this, kindly complete the following SURVEY. Thank you.
INTRODUCTION
Last week marked two decades since the launch of The Apple iPod which transformed how we listen to music. Its tagline was you can ‘hold 1,000 songs in your pocket’. It was an incredible success selling over 400m units. Read More. Yet the music industry continues to evolve with changing user needs, hence why we are exploring Spotify.
Spotify provides access to every single music produced, but how do you ‘sift out of the flood’ and find and discover new music that aligns with your taste?
That is the subject of this article on how Spotify has harnessed Machine Learning to create a powerful music recommendation engine. It is a similar concept to how Amazon recommends books to you after your checkout, and Netflix recommends movies.
Note: I’ll use this symbol anytime I am explaining a technical concept — [tech]…[/tech]
Discovery and New Users
Spotify initially focused on building a product where a user simple searches for a song in a search bar, finds it, and listens. This gave music fan users (who already knew what they wanted to listen to) the perfect listening experience. The Spotify team had done this because they could relate to these users.
If Spotify wanted to continue to grow, it needed to bring more casual listeners onto the platform, through recommendations. This brings us to lesson 1.
Lesson #1: Build for yourself first. But don’t build for yourself only.
“Designing with yourself in mind is a good place to start, but don’t limit your potential to people just like you”.
As recommendations was secondary at the time, Spotify dedicated little resources to it and only did a bit of Collaborative filtering, and outsourced recommendations to a start-up called The Echo Nest.
[tech] — What is Machine Learning?
Machine learning is simply when a computer (algorithm) is used to make predictions . Based off its input data, it would identify the pattern in the data, and with access to more data become even more accurate. Hence it is learning, much like humans.
It is applied in Speech Recognition (Alexa), Customer Service (Chatbots), Computer Vision (photo tagging in social media), and recommendation engines (Amazon, Netflix, and Spotify).
[/tech]
[tech] — What is Collaborative Filtering?
Collaborative filtering is when a large group of users put the same set of tracks together, on the same type of playlist continuously. This indicates that those tracks go well together, and those tracks are similar. An algorithm then uses that information to determine the similarity between the two tracks.
[/tech]
By 2011, the macro wind (mentioned in the previous episode) was shifting from curation-focused to recommendation-focused services. This made it a priority for Spotify and other players in the industry.
Spotify’s advantage was its library database of hundreds of million of playlists already created. But it was very difficult to work with such a massive database. There was a lot of noise in the data because many users’ playlist tracks together that weren’t similar.
So, they could have Christmas music and Pop together, so when you play Pop music, it plays Christmas carols after, offering a broken listening experience.
This brings us to lesson 2.
Listening to 🎵 Nostálgico by Rvssian, Rauw Alejandro & Chris Brown
Lesson #2 — Throwing Machine Learning at data you don’t fully understand isn’t enough to give you a great product.
Collaborative Filtering showed which playlist and tracks go with each other, but the team knew nothing about the tracks. To use machine learning effectively, you need to understand your data deeply.
When you start using Machine Learning it is easy to focus on applying it to your data, to make prediction of future events, but the ultimate goal should be ‘to create true ‘Machine Learning-first’ products’.
Take online shopping for example:
“As machine learning gets more and more accurate, an online store could move from just slowly improving its existing ‘people-who-bought-this-also-bought-that’ recommendations to upending the entire shopping experience.”
“When the accuracy reaches say 8 out 10 perfect predictions for what you will buy, the business model could change, and instead of waiting for you to place your order on those 8 items every week, the online store could ship you boxes of the 10 items it predicts you will want, and then you “shop” in the comfort and convenience of your own home by choosing which items to keep and send the rest back.”
“This is the move from ‘shopping, then shipping’ to ‘shipping, then shopping’.”
To solve this challenge for Spotify required two parts which I’ll call Qualitative and Quantitative.
The Qualitative part involved understanding how Artist and their music were described by listeners, at scale. While the Quantitative part involved the measured aspects of songs like the beat and frequency, at scale.
The Startup (Echo Nest) was working on the Qualitative part by using machine learning algorithms and web crawlers to review blogs, and the entire internet, to identify the words which every artist was described by and associate them with that word. The technical term for this technique is Natural Language Processing. The Quantitative part involved analyzing listening data by looking at the sound signal of the music.
“In other words, The Echo Nest had what we lacked — algorithms that said everything about the music itself but nothing about how listeners interacted with it — and Spotify had what they lacked: listening data on how people interacted with the music.”
This brings us to Lesson 3.
Lesson #3 — If you don’t have one side of an equation inside the company — look outside for it.
So, in 2014, Spotify acquired Echo Nest, and launched their 1st Machine Learning (ML) first product: Discover Weekly.
“This was Spotify’s first fully algorithmically generated playlist, individualized for every user.”
By giving users a weekly playlist where they could save the track they liked, Discover Weekly enabled Spotify go from ‘shopping then shipping’, to ‘shipping then shopping’.
Discover Weekly was a success.
But something unexpected happened. The ‘die-hard’ music fans loved it, but the mainstream fans were not impressed.
This was because the die-hard fans used it regularly, and so the algorithm’s data was biased towards them and hence recommended playlists that they liked based on this bias.
To overcome this bias, Spotify would need to refine its algorithm with human playlist curators.
Human + Machine: Algotorial
Spotify also acquired a music app, Tunigo. While the rest of the industry described the song by scientific terms, the Tunigo editors assembled soundtracks of what users can do with the music. Playlist like ‘Deep Focus’, and ‘High energy Running’.
This made it easy for anyone to navigate the music catalogue.
Spotify combined the solutions from both startups and tested the algorithm on a small set of users. This delivered a personalized listening session and achieved incredible success. Engagement increased by 200%.
This brings us to our final lesson.
Lesson #4 — In a machine l earning world, you have to learn the product, not build it!
This thinking was pioneered by the famous Computer Science professor at Stanford: Andrew Ng. To explain let’s consider how digital products are built.
Building Products
“Previously, a product manager would specify the product to be built in a product requirements document (PRD), and a wireframe to visualize it. This is then handed over to the engineers to develop it.”
“But if you are trying to build a machine learning product like an e-commerce recommendation system ‘the test is the new wireframe’. Instead of producing a PRD or wireframe — the product managers should source some example features and uses those examples to see how the ‘test set’ matches their machine learning system.” — Andrew Ng.
This was exactly what the Echo Nest and Tunigo teams did.
Listening to 🎵 Pray by Victony
SUMMARY
Lesson #1: Build for yourself first. But don’t build for yourself only.
Lesson #2 — Throwing Machine Learning at data you don’t fully understand isn’t enough to give you a great product. You need to understand your data deeply.
Lesson #3 — If you don’t have one side of an equation inside the company — look outside for it. And weigh the present-day costs against how far ahead you can leapfrog into the future.
Lesson #4 — In a machine l earning world, you have to learn the product, not build it!
There you have it. Please don’t forget to complete the SURVEY for my dissertation and also share the link. Thank you.
I hope you found this informative and relevant.
You can catch up with past articles in this series: Part 0, Part 1, Part 2, Part 3, Part 4.
If you’re finding this newsletter valuable, consider sharing it with friends, or subscribing if you haven’t already.
Also, if you have any questions about any topic, you would like me to tackle, kindly email me at notesbynero@gmail.com .
You can also follow me on LinkedIn, and Instagram.
