Machine Learning, Editorial, Programming, Tutorial

Recommendation System Tutorial with Python using Collaborative Filtering

Building a machine learning recommendation system tutorial using Python and collaborative filtering for a Netflix use case.

Towards AI Team
Oct 13 · 12 min read

Author(s): Saniya Parveez, Roberto Iriondo

Introduction

A recommendation system generates a compiled list of items in which a user might be interested, in the reciprocity of their current selection of item(s). It expands users’ suggestions without any disturbance or monotony, and it does not recommend items that the user already knows.

For instance, the Netflix recommendation system offers recommendations by matching and searching similar users' habits and suggesting movies that share characteristics with films that users have rated highly.

In this tutorial, we will dive into building a recommendation system for Netflix.

This tutorial’s code is available on Github and its full implementation as well on Google Colab.

The recommendation system workflow shown in the diagram above shows the user’s collaboration regarding the ratings of different movies or shows. New users get their recommendations based on the recommendations of existing users.

According to McKinsey:

75% of what people are watching on Netflix comes from recommendations [1].

Netflix Real-time data cases:

  • More than 20,000 movies and shows.
  • 2 million users.

Complications

Recommender systems are machine learning-based systems that scan through all possible options and provides a prediction or recommendation. However, building a recommendation system has the below complications:

  • Users’ data is interchangeable.
  • The data volume is large and includes a significant list of movies, shows, customers’ profiles and interests, ratings, and other data points.
  • New registered customers use to have very limited information.
  • Real-time prediction for users.
  • Old users can have an overabundance of information.
  • It should not show items that are very different or too similar.
  • Users can change the rating of items on change of his/her mind.

Types of Recommendation Systems

There are two types of recommendation systems:

  • Content filtering recommender systems.
  • Collaborative filtering based recommender systems.

Fun fact: Netflix‘s recommender system filtering architecture bases on collaborative filtering [2] [3].

Content Filtering

Content filtering expects the side information such as the properties of a song (song name, singer name, movie name, language, and others.). Recommender systems perform well, even if new items are added to the library. A recommender system’s algorithm expects to include all side properties of its library’s items.

An essential aspect of content filtering:

  • Expects item information.
  • Item information should be in a text document.

Collaborative Filtering

The idea behind collaborative filtering is to consider users’ opinions on different videos and recommend the best video to each user based on the user’s previous rankings and the opinion of other similar types of users.

Pros:

  • It does not need a movie’s side knowledge like genres.
  • It uses information collected from other users to recommend new items to the current user.

Cons:

  • It does not achieve recommendation on a new movie or shows that have no ratings.
  • It requires the user community and can have a sparsity problem.

Different techniques of Collaborative filtering:

Non-probabilistic algorithm

  • User-based nearest neighbor.
  • Item-based nearest neighbor.
  • Reducing dimensionality.

Probabilistic algorithm

  • Bayesian-network model.
  • EM algorithm.

Issues in Collaborative Filtering

There are several challenges for collaborative filtering, as mentioned below:

Sparseness

The Netflix recommendation system’s dataset is extensive, and the user-item matrix used for the algorithm could be vast and sparse, so this encounters the problem of performance.

The sparsity of data derives from the ratio of the empty and total records in the user-item matrix.

Sparsity = 1 — |R|/|I|*|U|

Where,

R = Rating

I = Items

U = Users

Cold Start

This problem encounters when the system has no information to make recommendations for the new users. As a result, the matrix factorization techniques cannot apply.

This problem brings two observations:

  • How to recommend a new video for users?
  • What video to recommend to new users?

Solutions:

  • Suggest or ask users to rate videos.
  • Default voting for videos.
  • Use other techniques like content-based or demographic for the initial phase.

User-based Nearest Neighbor

The basic technique of user-based Nearest Neighbor for the user John:

John is an active Netflix user and has not seen a video “v” yet. Here, the user-based nearest neighbor algorithm will work like below:

  • The technique finds a set of users or nearest neighbors who have liked the same items as John in the past and have rated video “v.”
  • Algorithm predicts.
  • Performs for all the items John has not seen and recommends.

Essentially, the user-based nearest neighbor algorithm generates a prediction for item i by analyzing the rating for i from users in u’s neighborhood.

Let’s calculate user similarity for the prediction:

Where:

a, b = Users

r(a, p)= Rating of user a for item p

P = Set of items. Rated by both users a and b

Prediction based on the similarity function:

Here, similar users are defined by those that like similar movies or videos.

Challenges

  • For a considerable amount of data, the algorithm encounters severe performance and scaling issues.
  • Computationally expansiveness O(MN) can encounter in the worst case. Where M is the number of customers and N is the number of items.
  • Performance can be increase by applying the methodology of dimensionality reduction. However, it can reduce the quality of the recommendation system.

Item-based Nearest Neighbor

This technique generates predictions based on similarities between different videos or movies or items.

Prediction for a user u and item i is composed of a weighted sum of the user u’s ratings for items most similar to i.

Figure 7: Item-based nearest neighbor prediction. | recommendation system tutorial with Python using collaborative filtering
Figure 7: Item-based nearest neighbor prediction. | recommendation system tutorial with Python using collaborative filtering
Figure 7: Item-based nearest neighbor prediction.

As shown in figure 8, look for the videos that are similar to video5. Hence, the recommendation is very similar to video4.

Role of Cosine Similarity in building Recommenders

The cosine similarity is a metric used to find the similarity between the items/products irrespective of their size. We calculate the cosine of an angle by measuring between any two vectors in a multidimensional space. It is applicable for supporting documents of a considerable size due to the dimensions.

Where:

cosine is an angle calculated between -1 to 1 where -1 denotes dissimilar items, and 1 shows items which are a correct match.

cos p. q — gives the dot product between the vectors.

||p|| ||q|| — represents the product of vector’s magnitude

Why do Baseline Predictors for Recommenders matter?

Baseline Predictors are independent of the user’s rating, but they provide predictions to the new user’s

General Baseline form

Where,

bu and bi are users and item baseline predictors.

Motivation for Baseline

  • Imputation of missing values with baseline values.
  • compare accuracy with advanced model

Netflix Movie Recommendation System

Problem Statement

Netflix is a platform that provides online movie and video streaming. Netflix wants to build a recommendation system to predict a list of movies for users based on other movies' likes or dislikes. This recommendation will be for every user based on his/her unique interest.

Netflix Dataset

  • combine_data_2.txt: This text file contains movie_id, customer_id, rating, date
  • movie_title.csv: This CSV file contains movie_id and movie_title

Load Dataset

Analysis of the Dataset

Find duplicate ratings:

Split train and test data:

Count number of ratings in the training data set:

Find the number of rated movies per user:

Find the Rating number per Movie:

Create User-Item Sparse Matrix

In a user-item sparse matrix, items’ values are present in the column, and users’ values are present in the rows. The rating of the user is present in the cell. Such is a sparse matrix because there can be the possibility that the user cannot rate every movie items, and many items can be empty or zero.

User-item Train Sparse matrix

User-item test sparse matrix

Global Average Rating

Check the Cold Start Problem

Calculate the average rating

Average Rating User

Average Rating Movie

Check Cold Start Problem: User

Here, 1% of total users are new, and they will have no proper rating available. Therefore, this can bring the issue of the cold start problem.

Check Cold Start Problem: Movie

Here, 20% of total movies are new, and their rating might not be available in the dataset. Consequently, this can bring the issue of the cold start problem.

Similarity Matrix

A similarity matrix is critical to measure and calculate the similarity between user-profiles and movies to generate recommendations. Fundamentally, this kind of matrix calculates the similarity between two data points.

In the matrix shown in figure 17, video2 and video5 are very similar. The computation of the similarity matrix is a very tedious job because it requires a powerful computational system.

Compute User Similarity Matrix

Computation of user similarity to find similarities of the top 100 users:

Compute Movie Similarity Matrix

Load movies title data set

Compute similar movies:

Get a similar movies list:

Building the Machine Learning Model

Create a Sample Sparse Matrix

Sample Sparse Matrix for the training data:

Sample Sparse Matrix for the test data:

Featuring the Data

Featuring is a process to create new features by adding different aspects of variables. Here, five similar profile users and similar types of movies features will be created. These new features help relate the similarities between different movies and users. Below new features will be added in the data set after featuring of data:

Featuring (adding new similar features) for the training data:

Featuring (adding new similar features) for the test data:

Training and Prediction of the Model

Divide the train and test data from the similar_features dataset:

Utility method to check accuracy:

Fit to XGBRegressor algorithm with 100 estimators:

Predict the result of the test data set:

Check accuracy of predicted data:

As shown in figure 24, the RMSE (Root mean squared error) for the predicted model dataset is 0.99.

Plot Feature Importance

Feature importance is an important technique that selects a score to input features based on how valuable they are at predicting a target variable.

The plot shown in figure 25 displays the feature importance of each feature. Here, the user_average rating is a critical feature. Its score is higher than the other features. Other features like similar user ratings and similar movie ratings have been created to relate the similarity between different users and movies.

Conclusion

Over the years, Machine learning has solved several challenges for companies like Netflix, Amazon, Google, Facebook, and others. The recommender system for Netflix helps the user filter through information in a massive list of movies and shows based on his/her choice. A recommender system must interact with the users to learn their preferences to provide recommendations.

Collaborative filtering (CF) is a very popular recommendation system algorithm for the prediction and recommendation based on other users’ ratings and collaboration. User-based collaborative filtering was the first automated collaborative filtering mechanism. It is also called k-NN collaborative filtering. The problem of collaborative filtering is to predict how well a user will like an item that he has not rated given a set of existing choice judgments for a population of users [4].

DISCLAIMER: The views expressed in this article are those of the author(s) and do not represent the views of Carnegie Mellon University, nor other companies (directly or indirectly) associated with the author(s). These writings do not intend to be final products, yet rather a reflection of current thinking, along with being a catalyst for discussion and improvement.

Published via Towards AI

Resources:

Google Colab implementation.

Github repository.

References:

[1] How retailers can keep up with consumers, McKinsey & Company, https://www.mckinsey.com/industries/retail/our-insights/how-retailers-can-keep-up-with-consumers

[2] How Netflix’s Recommendation System Works, Netflix Research, https://help.netflix.com/en/node/100639

[3] Recommendations, Figuring out how to bring unique joy to each member, Netflix Research, https://research.netflix.com/research-area/recommendations

[4] Collaborative Filtering, University of Pittsburgh, Peter Brusilovsky, Sue Yeon and Danielle Lee, https://pitt.edu/~peterb/2480-122/CollaborativeFiltering.pdf

Towards AI — Multidisciplinary Science Journal

The Best of Tech, Science, and Engineering.

Sign up for Towards AI Newsletter

By Towards AI — Multidisciplinary Science Journal

Towards AI publishes the best of tech, science, and engineering. Subscribe with us to receive our newsletter right on your inbox. For sponsorship opportunities, please email us at pub@towardsai.net Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Towards AI Team

Written by

Publishing the Best of Tech, Science, and Engineering | For Authors→ https://towardsai.net/contribute | Subscribe→ https://towardsai.net/subscribe — @Towards_AI

Towards AI — Multidisciplinary Science Journal

Towards AI is a world’s leading multidisciplinary science publication. Towards AI publishes the best of tech, science, and engineering. Read by thought-leaders and decision-makers around the world.

Towards AI Team

Written by

Publishing the Best of Tech, Science, and Engineering | For Authors→ https://towardsai.net/contribute | Subscribe→ https://towardsai.net/subscribe — @Towards_AI

Towards AI — Multidisciplinary Science Journal

Towards AI is a world’s leading multidisciplinary science publication. Towards AI publishes the best of tech, science, and engineering. Read by thought-leaders and decision-makers around the world.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store