Metadata-based Recommender Systems in Python

This blog illustrates a metadata-based recommender system in python

Before starting with the implementation of Metadata-Based Recommender systems in python, I will recommend you to give a short 4-min read to this blog which defines a recommender system and its types in laymen terms.

https://medium.com/@saketgarodia/the-world-of-recommender-systems-e4ea504341ac?source=friends_link&sk=508a980d8391daa93530a32e9c927a87

Through this blog, I will show how to implement a Metadata-based recommender system in Python on Kaggle’s MovieLens 100k dataset.

The dataset we will be using is the MovieLens 100k dataset on Kaggle :

MovieLens 100K Dataset

Let us start implementing it.

Problem formulation

To build a recommender system that recommends movies based on the genre, cast, crew and some keywords of a previously watched movie.

Implementation

First, let us import all the necessary libraries that we will be using to make a content-based recommendation system. Let us also import the necessary data files.

#importing necessary libraries
import numpy as np
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
from ast import literal_eval
from sklearn.feature_extraction.text import CountVectorizer
movies = pd.read_csv(‘movies_metadata.csv’)
credits = pd.read_csv(‘credits.csv’)
#keywords like jealousy, fishing, etc that belongs to particular movies are also part of the metadata.
#we will grab keywords from keywords.csv
keywords = pd.read_csv(‘keywords.csv’)
#importing necessary columns
movies = movies[[‘id’,’title’,’genres’]]
#clean movie_id function
def clean_id(x):
try:
return int(x)
except:
return np.nan
movies['id'] = movies['id'].apply(clean_id)
movies = movies[movies['id'].notnull()]
#converting everything into integer
movies['id'] = movies['id'].astype('int')
keywords['id'] = keywords['id'].astype('int')
credits['id'] = credits['id'].astype('int')
#merging the 3 dataframes to get all the required data on 1 datafarame movies
movies = movies.merge(credits, on='id')
movies = movies.merge(keywords, on='id')

After merging the independent data files credits.csv, keywords.csv and movies_metadata.csv, here’s what we have in the movies data frame:

Now, let us do the required wrangling of the data because the data seem to be in a messy format with a lot of information we won’t be using in our analysis.

The genres, cast, crew, and keywords columns all are of an object (or a string datatype). Let us get the required words we will be using from these columns by first using literal_eval to convert these strings into python objects (a list of dictionaries here) and use pandas and numpy to wrangle them.

#changing the 4 columns into python objects ( list of dictionaries here)
movies[‘genres’] = movies[‘genres’].apply(literal_eval)
movies[‘cast’] = movies[‘cast’].apply(literal_eval)
movies[‘crew’] = movies[‘crew’].apply(literal_eval)
movies[‘keywords’] = movies[‘keywords’].apply(literal_eval)
#grabbing the names of all the genres attached to each movie
movies[‘genres’] = movies[‘genres’].apply(lambda x: [i[‘name’].lower() for i in x])
#grabbing the name of the director from all the crew members
#we will only use directors from the creqw column for our purpose
movies[‘crew’] = movies[‘crew’].apply(lambda x: [i[‘name’].lower() for i in x if i[‘job’]==’Director’])
#grabbing the cast and keywords from the list of dictionaries of those columns
movies[‘cast’] = movies[‘cast’].apply(lambda x: [i[‘name’].lower() for i in x])
movies[‘keywords’] = movies[‘keywords’].apply(lambda x: [i[‘name’].lower() for i in x])
#taking maximum 3 cast/genre/keywords for each movie
movies[‘genres’] = movies[‘genres’].apply(lambda x: x[:3] if len(x)>3 else x)
movies[‘cast’] = movies[‘cast’].apply(lambda x: x[:3] if len(x)>3 else x)
movies[‘keywords’] = movies[‘keywords’].apply(lambda x: x[:3] if len(x)>3 else x)

After performing the necessary cleaning and choosing just 3 genres, 3 keywords and cast for illustration purpose, here’s how our data looks:

Now, we have the required clean data for building metadata-based recommender systems. We just need to remove spaces between the names and surnames because if we won’t remove the spaces, movies for Tom Cruise and Tom Hanks will be considered the same by machine because the first name ‘Tom’ is common. Let us remove the spaces so that Tom Hanks becomes TomHanks and Tom Cruise becomes TomCruise.

#removing spaces
movies[‘cast’] = movies[‘cast’].apply(lambda x: [i.replace(‘ ‘,’’) for i in x])
movies[‘crew’] = movies[‘crew’].apply(lambda x: [i.replace(‘ ‘,’’) for i in x])
movies[‘keywords’] = movies[‘keywords’].apply(lambda x: [i.replace(‘ ‘,’’) for i in x])
movies[‘genres’] = movies[‘genres’].apply(lambda x: [i.replace(‘ ‘,’’) for i in x])

Now, all the names in the cast have 0 spaces which makes them unique.

Now, let us make 1 column of all the metadata that we have by appending the values in the genres, cast, crew, and keywords column.

movies[‘metadata’] = movies.apply(lambda x : ‘ ‘.join(x[‘genres’]) + ‘ ‘ + ‘ ‘.join(x[‘cast’]) + ‘ ‘ + ‘ ‘.join(x[‘crew’]) + ‘ ‘ + ‘ ‘.join(x[‘keywords’]), axis = 1)

Due to memory issues in google colab, I will just run the first 10000 movies to build a recommender system. The same codes can be used to scale further.

We will use a CountVectorizer to built numeric features from our metadata. We won’t use TfIdf here because there might be many movies with the same directors and we definitely don’t wanna penalize that director. It might be possible that a user wants to be recommended movies belonging to that director. Most of the words we have are names and genres whose counts are actually useful for recommending movies.

I will use cosine similarity to find the similarity between any 2 movies. Now let's make a cosine similarity matrix using count vectorizer values and then let's build a recommender function.

count_vec = CountVectorizer(stop_words=’english’)
count_vec_matrix = count_vec.fit_transform(movies_df[‘metadata’])
cosine_sim_matrix = cosine_similarity(count_vec_matrix, count_vec_matrix)
#movies index mapping
mapping = pd.Series(movies_df.index,index = movies_df[‘title’])

#recommender function to recommend movies based on metadata
def recommend_movies_based_on_metadata(movie_input):
movie_index = mapping[movie_input]
#get similarity values with other movies
similarity_score = list(enumerate(cosine_sim_matrix[movie_index]))
similarity_score = sorted(similarity_score, key=lambda x: x[1], reverse=True)
# Get the scores of the 15 most similar movies. Ignore the first movie.
similarity_score = similarity_score[1:15]
movie_indices = [i[0] for i in similarity_score]
return (movies_df[‘title’].iloc[movie_indices])

Now that we have built our recommender function, let's see how it works. Let’s try to get movies similar to ‘Blind Horizon’ using metadata-based recommender system.

recommend_movies_based_on_metadata(‘Blind Horizon’)

For the input movie ‘Blind Horizon’, we have been recommended 15 movies by learning its metadata. Isn’t it amazing?

To know about the Content and Collaborative-Filtering based approaches, go through my following blogs:

1. Content-based Recommender Systems: https://medium.com/@saketgarodia/content-based-recommender-systems-in-python-2b330e01eb80?
2. Recommender Systems using Collaborative Filtering: https://medium.com/@saketgarodia/recommendation-system-using-collaborative-filtering-cc310e641fde

Thank you

Keep learning