Recommending a Movie Using Collaborative Filtering

ALSO, ARE RECOMMENDER SYSTEMS INFLUENCING OUR TASTE??

An excerpt on creating a movie recommender system similar to the OTT platforms.

Photo by Georgia Vagim on Unsplash

INTRODUCTION

Formally Defining,A Recommender System is a system that seeks to predict or filter preferences according to the user’s preferences. The demand for a good recommender system is soaring, especially with then onset of Covid-19 induced lock down,forcing everyone to stay home and watch movies of their favourite genre,actor,director….you get it right.This is where a recommender system plays an important role in providing the user, content he is more likely to watch, rather than the user searching for something that interests him,which would mess with the user experience.

The essence of a recommender system lies in its recommendation engine.There are Two types of Recommendation engine:

1. Content-based filtering engine: It provides recommendations by matching the description of the movie and a user profile, generated by the interests provided by the user.It has an explicit understanding of the recommendation.You might have observed it in some apps,where you are asked questions about your preferences as soon as you signup.This is what it’s for.
2. Collaborative filtering engine: It is a method of making automatic predictions about the interests of a user by collecting preferences or taste information based on the activity of current user along with many other users with similar activity(collaborating).The underlying assumption of the collaborative filtering approach is that if a person A has the same opinion as a person B on an issue, A is more likely to have B’s opinion on a different issue than that of a randomly chosen person.It need not have any explicit understanding of the recommendation.You might have observed in one of your OTT platforms when you open a particular movie, An array of movies under the heading “people who watched this movie also watched”.This is what it uses.

Equipped with this basics,Lets dive into creating a movie recommender system using collaborative filtering.

We start by Importing required libraries. We will be using Scikit-surprise which contains the SVD(Singular Value Decomposition).SVD allows us to extract and untangle information,which is really helpful in creating a recommender system.

This topic involves a lot of statistical data analysis.resources to know more about scikit surprise,SVD:

Welcome to Surprise' documentation! - Surprise 1 documentation

Matrix Factorization-based algorithms - Surprise 1 documentation

First thing one must do before creating a model is observe the data. This gives us a lot of insight on the type of data it is, and what we could use to gain the maximum from it.

As we observe the data, we see that timestamp is a redundant column and it is best to remove it.

It is always a good practice to check for NaNs in your dataset,luckily we don’t have any.

Now comes the Main Part of this model, Exploratory Data Analysis

To start,We look for the Number of movies and users in the dataset.

Now we find Sparsity of the data. Sparsity tells us the percentage of movies missing rating by the users. i.e Not all users rate a movie, It tells us the percentage of missing values by the total values.Sparsity for this data is 98%. Usually the lower the sparsity,the better.But in the case of Collaborative Filtering, below 99% is manageable.

Sparsity(%) = (No of Missing Values/(Total Values))*100

Now we try to visualize ratings distribution.

Most of the ratings are between 3–5 and the range of the ratings are from 0.5 to 5.

FEATURE ENGINEERING

Now comes The next essential part of the system, Feature Engineering.I always believe that Feature Engineering as Important as building a model, as It allows the model to better understand and converge better.

Here We are Reducing the Dimensions by removing the redundant data like Movies with less than 3 ratings or user who rated less than 3 movies, as it is difficult to recommend something with such less data to analyse.

Now lets start creating the Model,

Creating a Surprise Dataset for training using the Reader class that we imported and provide the expected scale of rating,which we found out during our exploratory data analysis.You can add that to your data using the dataset import.

Now as we are using our whole train set for training,we create an antiset which consists of all the data without the reviews on which we can test.

We create our SVD, which untangles the information for us to complete the recommender model.

We then evaluate our model with the metrics Root Mean Square Error and Mean Absolute Error as they provide the average over the epoch of the absolute values of difference between the recommendation and the actual observation.

Predicting

The prediction gives us a movie id for user id 1.

This finishes our recommender system’s job.

Now… lets discuss about something debatable.

Are Recommender Systems influencing our taste in movies and taking the control from us??

Photo by Juan Rumimpunu on Unsplash

My Father who is no way related to computer Science asked me this one fine morning.He was going through his favourite video streaming service and made an observation that, He was seeing videos that are related to a few areas only. It made him feel that his choice is getting Influenced by it and was unable to come across something new.

I explained this to him using my own words and understanding:

He has been watching the same videos over and over daily,Thus creating a profile that, he is interested in only in this particular topic of videos.That was the reason he was shown videos from that particular topic only.

But does it mean you have no control over it,

The Answer is NO.

You still have your control, If you are not interested in a topic, but you were recommended by the engine, Just let the engine know that you are not interested. Yes, you have that option. Expand your viewing horizons for diverse content. A recommender system is there just to help you, not control you.It all finally depends on the viewer to watch or not.

Lets share our views on this and spread some knowledge.Lets learn and grow as a community.. Because all we are left with is people,memories and knowledge.

Thank you.