It’s Show Time! But wait. What shall I watch now?
We’ve all been there, I believe. Wanting to watch something, but not knowing exactly what to watch … Wait, I think you have an option here! Well, there’s this series on Netflix that your friends have been talking about for a while. I know you’ve already watched the trailer and you don’t think you’ll like it that much, but shouldn’t you watch it anyway to get involved in their discussions? Go give it a shot!
You watched the pilot, but still …
NOT interesting! Why is everyone so hyped-up about it anyway?!
I totally understand, but before giving up, remember: your friends have told you that the story starts slowly, and then throughout the episodes, you get more and more addicted to it.
Alright Fouad, let’s see where this goes.
You finished the 8 episodes of the season — we don’t want to mention that you skipped like half of them, and still felt like these are the longest 8 episodes you’ve ever watched — and then you’re like:
Fine, fine! You feel that you lost like 8 episodes of your life. If you had known this in advance, of course, you wouldn’t have done it!
But, wait a minute … is that even an option? How can you possibly do that? How can you know if you would like something you will watch, before even watching it?
Here is where the Netflix Recommendation Engine (NRE) comes into play. NRE is composed of a set of AI algorithms that learn what you like and dislike through your interaction, and then use this knowledge to suggest what you most probably would enjoy watching next.
Questions we will be answering in this blog
- How does Netflix use AI to recommend what to watch next?
- How does Netflix use AI to recommend a show in the best possible way?
- What are other AI use cases that we might not be aware of in Netflix?
- Why AI specifically? What are the benefits that AI has brought to Netflix?
- What are the challenges that Netflix faces when using AI to operate on such a large scale of users?
See? We’ve got a lot to cover. But… First things first!
Uncovering the magic of Recommendations
Well, let’s start by saying that Netflix knows your taste better than any of your friends. Do you know why? It’s simply because the knowledge that Netflix has about you originates from YOU: your previously watched movies and series, your interaction while watching (e.g. skipping, pausing, repeating), the time you take to finish an episode, etc. This information is their way of understanding your taste and recommending what to watch next accordingly.
Seems a bit vague? Let’s clarify it through an example. Let’s imagine for a moment that Netflix has 3 users and 5 shows. These 3 users have rated (liked or disliked) the shows they watched as you can see in the figure below.
The problem that Netflix needs to solve is: Given the above matrix, what shall we recommend for a specific user to watch next? Here we have 2 approaches:
1. Content-Based Filtering
As a first intuition, someone might try to deduce the content that a person will like based on what he has watched before. So, if we take as an example user A, we can see that she is more into action stuff than she is in comedy, and as a result, the next recommendation would be something like Prison Break. That’s the intuition!
Here the process is simple, you know. A few shows … fewer users… but in real life, it’s a bit more complicated. What we really do in Content Filtering is characterize the content we have (the shows) according to specific features (like how much it involves comedy, how much it involves action, etc), and then recommend content that is similar to what the user usually likes. That’s one way of doing things.
2. Collaborative Filtering
Collaborative filtering is the other way around where recommendations are not only based on what the user has watched. They’re based on what other similar users have watched and enjoyed. And how’s that exactly? Well, instead of characterizing content according to specific features, we characterize the users themselves (like how much drama they enjoy, how much comedy they like watching, etc). This information is taken once the user signs up for Netflix and gets updated according to what he watches. Like this, we will be able to identify similar users that share a similar set of features, or in other words, a similar taste, and this similarity will help empower future recommendations.
The picture below summarizes the difference between Content-Based and Collaborative filtering. It’s good to know that Netflix uses a mix of these two approaches to ensure better recommendations.
See? The process is simple. One last detail to clear your thoughts is: how do we obtain the features that we talked about, and what are they exactly? Well, you should note here that these features are not known beforehand, they are learned, and that’s the role of Machine Learning (ML) in the process. Once these features are learned, we can represent each user and each show by a vector or what we call “an embedding” in an n-dimensional space. Representing users and shows in the space will help us group similar users and similar shows together (because they will be close to each other), and this enhances future recommendations substantially. To understand how these embeddings are learned, you need to know the idea of Matrix Factorization.
Another new terminology. What does it mean?
Okay. In SIMPLE terms, Matrix Factorization is the process of replacing a matrix with 2 different matrices whose multiplication will give us our original matrix back.
Oh, it’s easy trust me. But anyway, let’s take a step back. When you learned about factorization in young classes, they used to ask you to factorize x² — 1 as a product of 2 polynomials, remember?
Genius old you would have easily noticed that x² — 1=(x — 1)(x+1). See how easy that was? Now the same thing applies to matrices. So in our case, we have a rating matrix containing information about how users rated specific shows. What we want to do through Matrix Factorization is find from our original matrix 2 matrices, one for the users and one for the shows. The product of these 2 matrices is supposed to give us back (approximately) the original rating matrix where we have a (+1) for a like and a (-1) for a dislike. Let’s examine the following figure and everything will become clearer.
To get the matrices for the users and the shows, we use ML. The goal is to find matrices that would give us the least error between the result of their product and the real original matrix. In our small example, we used 2 features to represent a user or a show, but in real life, the number is way more than that. The process of finding the 2 matrices happens according to the traditional ML workflow. At first, we initialize them randomly, and then we keep updating them in a way that their product becomes as close as possible to the real rating matrix. Once we have the 2 matrices, their product will give us what we call the Predicted Rating Matrix as you can see in the figure below.
Ok Fine, but what did we achieve?
Excellent question! If you look at the predicted rating matrix that we obtained, you can see that it contains numbers. These numbers reflect how each user would rate a specific show, and as a result, they show how much a user is supposed to enjoy a specific show. So, this matrix is sufficient for future recommendations. Let’s look at user B to understand things more. We can see that user B has already watched Friends and Blacklist. So, at this point, we need to recommend a show from the remaining ones (La Casa De Papel, Prison Break, and HIMYM). If we look at the predicted rating matrix we derived, we can see that user B is supposed to rate HIMYM the highest, so that would be the next recommendation.
See? That’s the intuition behind it: we predict how much a user would like a show and accordingly see if we recommend it or not.
Now that we know how Netflix uses AI to know what to recommend, it’s time to see how they use AI to know how to recommend it in the best possible way.
Going From Simple Recommendations to Personalized thumbnails
As you know, Netflix gives you suggestions in the form of image thumbnails. I’m sure you have noticed that while watching a series, for example, there are several times where you see the thumbnail of this series change. That’s not new information for you I assume. But what you might not know is that these thumbnails you’re seeing, they’re not common to everyone! These are personalized for you! And so, the thumbnail you’re seeing at a specific time for a specific show is not the same as what any other Netflix user is seeing.
But … Why does Netflix do this?!
Well, it has been proven that thumbnails have a great effect on you watching a show or not. In fact, the thumbnail is the first thing that gets your attention — even before the title — and by the end of the day, the goal is not only to recommend. The goal is to recommend in a way that maximizes the chances of you watching the recommendation to ensure a great user experience. Since thumbnails are that important, Netflix goes through a specific process before deciding which ones are relevant to show you.
- A Netflix video is made out of thousands or hundreds of thousands of static frames.
- These frames are not all good to be shown as thumbnails, and this is why we perform Aesthetic Visual Analysis (AVA) on them to select the ones that are the most appealing. The selected frames are also called candidate frames.
- Each of these candidate frames is then annotated according to different factors: brightness, number of faces, skin tones, symmetry, level of blurring, level of nudity, etc. So, at this stage ML comes in handy to explain what a frame holds.
- According to the provided annotations, Netflix ranks the candidate frames and chooses the ones that are supposed to maximize the click rates for specific users.
- And finally, the frames with the highest ranks are provided to users in a personalized fashion.
So, I think now it’s clear how AI is shaping Netflix recommendations in many ways to guarantee the best possible user experience. But the use of AI does not end at this stage in Netflix!
AI use cases that we don’t clearly see as users
Besides recommendations and thumbnails personalization — that might be clear to us as end-users — there are other use cases where Netflix uses AI for better decision-making.
1. Stream Optimization
Netflix has more than 200 million subscribers around the world. With this huge number, it usually becomes a bit difficult to always offer the best streaming quality. But with the help of AI, Netflix can predict based on the users’ activities what would be the shows they will be watching. For this purpose, Netflix caches these shows ahead of time to servers that are close to the end-users. This way, subscribers would be able to stream them with high quality even during peak hours, and this is why the majority of the users don’t even notice the presence of this problem at all.
2. Pre-Production
As you know, Netflix produces originals. But in order to do this in a successful way, Netflix uses the data they collect about users and their preferences to produce something they most likely will enjoy. From this data, they can decide on the main actors, the places for shooting, the budget they will spend, the countries where the show would be available, etc. So, they need to optimize things in a way that guarantees the success of the show from one side and a great user experience from the other side.
3. Post-Production
After deciding on the production details and finishing the shooting process, Netflix uses AI to make sure that the filming and the quality of the scenes are as intended. The role of AI here is mainly to specify the scenes that require further attention while editing. This reduces the human effort and the amount of time that would’ve been spent if the scenes were checked manually.
Final Thoughts
After getting to see how much Netflix relies on AI and on data-driven solutions, you might be asking yourself:
Why go through all of this trouble?!
You’re right! In fact, AI solutions are costly on many levels. So why does Netflix insist on using AI through different stages? The short answer to this is “the business needs”. In fact, no company uses such AI solutions just because they’re cool, NO! The goal is to keep the customers engaged and happy as much as possible, and give them the best experience because this would result in a higher subscription rate and a lower cancellation rate. The way Netflix uses AI has proven itself to be worthy.
“80% of what people play on Netflix actually comes from the recommendation algorithm, so it’s a really big lever for Netflix, it’s a really important part of what we do.” — Aish Fenton, Netflix Director of Machine Learning, 2018
But as promising as this sounds, the process is not challenge-free!
Of course, this is something expectable. Applying AI to understand the taste of a huge number of users is not supposed to be perfect all the time. So it is true that most of the recommendations that Netflix gives are accurate, but this does not mean that it’s always accurate, and this is because the human taste by itself is not something very static. Our taste changes with time, and what we enjoy watching today might have been something we used to hate 5 years ago. But despite all of that, the recommendation system in Netflix overcomes these challenges as much as possible by learning and adapting to the diversity of the users and their different tastes, and that’s one essential reason for their proven success.
Conclusion
Now I’m sure you know how AI impacted the revolution at Netflix and how they’re using it to always keep their users satisfied. From this blog, you’ve got a better intuition on how Netflix learns about your taste to recommend a show that you enjoy, and most importantly, you understood that these recommendations are not random or common, as you might have thought at some point. You also got the chance to discover some hidden areas where Netflix is applying AI, and finally, you saw what benefits and challenges this AI usage brings to the table.
The takeaway from this blog
Since by now you know the challenging process that Netflix goes through to understand your taste, for the next time, when you’re wondering what you should watch, don’t ask your friends, don’t watch something you don’t like just to follow up with their discussions. Just grab your popcorn and go to Netflix! I’m sure they will have something better for you. Happy Watching :D
Don’t forget to support with a clap!
You can join our efforts at Zaka and help democratize AI in your city! Reach out and let us know.
To discover Zaka, visit www.zaka.ai
Subscribe to our newsletter and follow us on our social media accounts to stay up to date with our news and activities:
