Improving Collaborative Filtering with Dimensionality Reduction
There are two ways of using dimensionality reduction in recommender systems:
The first is creating latent factor models which reduce the dimensions of both users and items simultaneously, and produce a dense matrix, which can generate rating predictions. Tumas covers latent factor models at length in this article.
The second is an extension of collaborative filtering. It reduces either users or items to a lower dimensional space, depending on whether you are using user-based or item-based methods. In the case of user-based collaborative filtering, item space is reduced, and peer groups of users are formed using these reduced dimensions.
Like clustering, this technique reduces runtime of the peer group creation process. It also yields improvement in accuracy.
Reducing the Matrix
You have two options for matrix decomposition, SVD or PCA. Both will improve the efficiency of peer group formation by almost exactly the same amount, as we end up with a m x d matrix in both cases. However, because these d eigenvectors are from a similarity matrix for SVD and a covariance matrix for PCA, the actual results might vary. I encourage you to experiment and choose the best option. In this tutorial, however, we will use PCA.
First, replace the missing values with zeros, or the mean of the respective row. We’ll call this filled matrix Rf.
Then, from this completed matrix Rf, we construct an n x n matrix representing relationships between each of the items. Since we using the PCA variant of dimensionality reduction, we compute the covariance matrix S of Rf.
We then use eigendecomposition to find the eigenvalues ∑ and the eigenvectors P of S. We take the top d eigenvectors from P to form P_d. P_d is an n x d matrix, and if we dot it with the original m x n user-item matrix, we have an m x d matrix, or a matrix of m users with d dimensions. You can then calculate the pairwise similarity for each pair of users in this lower dimensional space.
Reducing Bias
Replacing the unspecified values with the mean of each column introduces significant bias to the creation of the similarity/covariance matrix, and thus the predictions.
This makes sense; say there exists a correlation between giving A Bug’s Life and Finding Nemo a high rating. You would assume the similarity would be high between those two items. But, the input is sparse, and the set of ratings that showcase this relationship is small in comparison to the set of unspecified ratings for these two movies. These unspecified ratings will be set to the mean rating of the user in question. This means whether a user thinks Finding Nemo is the greatest cinematic creation of all time, or if they think it’s climate activist propaganda, their rating for A Bug’s Life will be set to the mean if it is unspecified.
This might be a good estimation for other measures, but it creates a large amount of unhelpful information if we’re trying to find similarity and covariance. Without using another similarity measure in tandem, the best way to address this problem is to only use the specified ratings to compute the similarity/covariance matrix.
There is also bias in the creation of item-reduced matrix m x d, as it is the product of mean-filled user-item matrix and the top d eigenvectors of similarity matrix. We can use only the specified ratings in the user-item matrix by using each user’s average contribution to each of the d eigenvector s. This creates an m x d matrix, as before.
Where is the set of all items user u has rated and e_ji is the jth value of the ith eigenvector (out of d).
We have a user-item matrix with reduced item dimensionality, just before, but now the bias estimation is diminished because it was created only with specified ratings.
So there we have it; another technique to improve the accuracy and efficacy of collaborative filtering, and two ways to reduce bias within it. We use many of the same dimensionality reduction techniques in our implementation of model-based collaborative filtering.
