Non Personalised Recommender System in Python
INTRODUCTION
In today’s world, recommender systems are the heart of almost every online platform. It is acting like a ghost sitting silently and influencing our most of the day to day decisions. Be it an online retail platform like AMAZON or movie service provider like NETFLIX, everybody relies a lot on recommender system to plod their customers to explore more products and ultimately improve the company’s profit.
The biggest problem faced by the internet user is the overload of information. If we look at the past two-three decades, there has been an exponential growth in the amount of information available online. This huge pile of information, although marked a revolutionary change in the knowledge gathering capability of human beings, made it quite difficult to manoeuvre through the webs of internet and find relevant and reliable information. This is when the industry realised the need of recommender systems which can filter, prioritise and efficiently deliver relevant information to the user in a very short period of time.
Recommender system, in its most naïve definition, is a system that provides a user with the most relevant information that the user is seeking on the basis of user’s preferences, interest, or observed behaviour about item. Initially, recommender systems started by recommending most popular or famous items and with the course of time developed into more sophisticated recommender systems utilising collaborative filtering, content-based filtering and subsequently hybrid filtering.
To provide comprehensive knowledge of the recommender systems, I will be posting a series of different recommender systems with their respective merits and demerits along with the Python Code. Please note that this series covers the exercises from University of Minnesota’s Recommender system specialisation courses on Coursera. While I can explain some of the concepts involved in these exercises along the way, it’s impossible for me to convey all the information you might need to fully comprehend it. If you’re really interested in recommender system but haven’t been exposed to it yet, I encourage you to check out this specialisation course. You can audit the course to access all the videos and course material for free. With that let’s get started!
In this section we will be looking at the most basic form of recommender system which is Non Personalised Recommender System.
Non Personalized Recommender System
Let’s start with an intuition of what Non personalized recommender system is. As the name suggests, this recommender system provides with general recommendations to the user without any context of what user wants or what is his preference. Let’s think of Amazon for example. When we visit the website, it provides us with a list of most popular item. These popular items can be based on different parameters like geography, age, sex etc. Based on these parameters, system calculates the mean of the product rating of all the users and provides us with the products with maximum mean. This is also called stereotyped recommender system.
But there is a problem in this approach of taking simple mean of the ratings. When there are very less number of ratings, we have a very less confidence of whether a good rating provided to an item is actually good or not. We do not want to be very much sanguine about an item being very good based on very few ratings. To overcome this challenge, we damp the overall mean. In this, we assume that without evidence, every item is average and every additional rating is the evidence if the given item is above average or below average. In simple language, it will not allow any item to have a very good rating on the basis of only few good ratings until sufficient number of good ratings is not provided.
To achieve this, we modify the formula of mean in the following way :
α : Strength of the evidence required to overcome the overall mean(Generally equals to 5 but can be changed as per business requirements)
µ : Global mean rating
What this K term will do is that when ther are very less number of ratings, it will damp some extreme positive ratings and as the number of rating increases, this damping effect will start decreasing and true mean will start coming into the picture.
Below is the code to implement Damped mean Recommender System. Click Here to download the Data Set. Please note that I am using the same data set provided in the course for the programming assignment.
The representation of the code below might not be very easy to read, so please go to my GitHub repository to access all the codes of Recommender Systems of this series.
import pandas as pd
import numpy as npRatings=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\ratings.csv”)
Movies=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\movies.csv”)
Tags=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\tags.csv”)
Ratings_mean=Ratings.groupby([‘movieId’])[[‘rating’]].mean().rename(columns = {‘rating’: ‘Mean_rating’}).reset_index()
# Calculating damped mean using alpha = 5
Ratings_sum=Ratings.groupby([‘movieId’])[[‘rating’]].sum().rename(columns = {‘rating’: ‘sum_rating’}).reset_index()
Ratings_sum[‘sum_rating_factor’]=Ratings_sum[‘sum_rating’]+5*(Ratings[“rating”].mean())
Ratings_count=Ratings.groupby([‘movieId’])[[‘rating’]].count().rename(columns = {‘rating’: ‘count_rating’}).reset_index()
Ratings_count[‘count_rating_factor’]=Ratings_count[‘count_rating’]+5
Ratings_damped=pd.merge(Ratings_sum,Ratings_count[[‘movieId’,’count_rating’,’count_rating_factor’]],on=[‘movieId’],how=’left’)
Ratings_damped[‘damped_mean’]=Ratings_damped[‘sum_rating_factor’]/Ratings_damped[‘count_rating_factor’]
Ratings_mean_dampmean=pd.merge(Ratings_mean[[‘movieId’,’Mean_rating’]],Ratings_damped[[‘movieId’,’damped_mean’]],on=[‘movieId’],how=’left’)
# Sorting to get top rated movies
Ratings_mean_dampmean = Ratings_mean_dampmean.sort([‘Mean_rating’], ascending=False)
I will encourage to see the difference between the ratings after the addition of the damping term.
Apart from Stereotyped Recommender System, another class of non personalised recommender system use association to recommend products. These are known as Product Association Recommenders.
In Product Association Recommenders, although not personalised, uses the knowledge of the product that a consumer is looking at and based on this knowledge, it provides item recommendations. One way of providing these recommendations is to go through all the historical transactions and see what other people have most frequently bought along with the given product in the same sales transaction.
This type of recommendation is not personalized to the person but to current user who is looking at the given product. So how to calculate this?
We can start with calculating what percentage of people who buy product X (product that the user is currently looking at) also buy another product Y in the same cart?
So we can count the number of people who buy X and Y, our numerator, divide it by the number of people who buy X. And if, say, 80% of people buy X and Y, then Y is a likely thing to add if you’re already buying X. It’s likely, but is it useful? Let us take an example to understand this. Let us suppose that X is Red sauce and Y is Banana. So is it correct saying that 80% of the people who buy Red sauce also buy Banana necessarily? Or does it mean that Banana is a very common product and people will buy it irrespective of the fact whether there is Red sauce in the basket or not.
To control the popularity of Y overwhelm the recommendation, we use the concept of Baye’s Theorem(Click here for detailed info). What this tells is that it describes the probability of an event, based on prior knowledge of conditions that might be related to the event. Moving forward in this direction, we will calculate the Probability of Y given X i.e. P(Y|X) divided by probability of Y alone i.e. P(Y).
Another way of calculating the association between two products is to calculate the Lift value. Lift is the probability that two items will be bought together divided by the product of their individual probability.
Below is the code to implement Basic association rule Recommender System.
import pandas as pd
import numpy as npRatings=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\ratings.csv”)
Movies=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\movies.csv”)
Tags=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\tags.csv”)movie_association = pd.DataFrame(columns=[‘movieId’,’movieId1',’association’])
distinct_movies=np.unique(Ratings[‘movieId’])
i=1
for movie in distinct_movies[157:158]:
movie_data=Ratings[Ratings[‘movieId’]==movie]
print “movie: “, i , “out of: “, len(distinct_movies)
j=1
for movie1 in distinct_movies:
movie1_data=Ratings[Ratings[‘movieId’]==movie1]
distinct_cust=len(np.unique(movie_data[‘userId’]))
movie2_data=movie1_data[movie1_data[‘userId’].isin(np.unique([movie_data[‘userId’]]))]
distinct_cust_common=len(np.unique(movie2_data[‘userId’]))
fraction_common_cust=float(distinct_cust_common)/float(distinct_cust)
movie_temp=pd.DataFrame(columns=[‘movieId’,’movieId1',’association’])
movie_temp = movie_temp.append({
“movieId”: movie,
“movieId1”: movie1,
“association”: fraction_common_cust
}, ignore_index=True)
movie_association = movie_association.append(movie_temp, ignore_index=True)
if j%500==0:
print j
j=j+1
i=i+1
movie_association=movie_association.sort_index(by=[‘movieId’,’association’]).reset_index()
Below is the code to implement Lift based Recommender System.
import pandas as pd
import numpy as npRatings=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\ratings.csv”)
Movies=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\movies.csv”)
Tags=pd.read_csv(“C:\Users\Ankur.Tomar\Desktop\Courses\Recommender\Non Personalized\\tags.csv”)movie_association = pd.DataFrame(columns=[‘movieId’,’movieId1',’association’])
distinct_movies=np.unique(Ratings[‘movieId’])
i=1
for movie in distinct_movies[1284:1285]:
movie_data=Ratings[Ratings[‘movieId’]==movie]
print “movie: “, i , “out of: “, len(distinct_movies)
j=1
for movie1 in distinct_movies:
movie1_data=Ratings[Ratings[‘movieId’]==movie1]
distinct_cust=len(np.unique(movie_data[‘userId’]))
distinct_cust1=len(np.unique(movie1_data[‘userId’]))
movie2_data=movie1_data[movie1_data[‘userId’].isin(np.unique([movie_data[‘userId’]]))]
distinct_cust_common=len(np.unique(movie2_data[‘userId’]))
fraction_common_cust=(float(distinct_cust_common)*len(np.unique(Ratings[‘userId’])))/(float(distinct_cust)*float(distinct_cust1))
movie_temp=pd.DataFrame(columns=[‘movieId’,’movieId1',’association’])
movie_temp = movie_temp.append({
“movieId”: movie,
“movieId1”: movie1,
“association”: fraction_common_cust
}, ignore_index=True)
movie_association = movie_association.append(movie_temp, ignore_index=True)
if j%500==0:
print j
j=j+1
i=i+1
movie_association=movie_association.sort_index(by=[‘movieId’,’association’]).reset_index()
In the next article, we will be looking at a bit more sophisticated form of recommender system called Content Based Recommender System.
Please go to my GitHub repository to access all the codes of Recommender Systems of this series.
Thanks!
