Managing Biases in Recommender Systems
Back when I was building my first recommender system for an e-commerce company, I was excited to try out all the state-of-the-art deep learning models. The model training went well. It was pure pleasure watching all those metrics going up, including NDCG, MRR, MAP, etc. However, the joy quickly dissipated when I found out that the outputs were awfully homogeneous — it was recommending a handful of top most popular items to almost all the users.
Of course, today, we know this is the popularity bias. Recommending only most popular items to everyone is an easily attainable local minimum that achieves “good” precision, hit rate, etc. When not handled properly, you can be wasting expensive GPU time training useless models, or even just re-training over and over again praying that the sacred random number generator outputs a weight initialization that favors the global minimum over the local minimum.
In this article, let’s take a look at how we can handle different kinds of biases — not just popularity bias — that commonly pop out in a recommender system.
Popularity Bias
Why Is This a Problem?
Serendipity is desirable in a good recommender as it helps user discover new interests or niche products and generates strong customer satisfaction. A popularity-biased recommender is harmful to the company as niche products cannot get the right exposure, while also imposing risk of losing customers who are actually seeking niche products.
How Can We Handle It?
Popularity bias originates from the model. There are a number of ways to handle it, including using regularization-based method [1] or a list diversification post-processing technique [2].
Due to niche products receiving less interactions and feedback from users, their scoring will likely be less accurate, no matter which technique you employ.
Drawing inspiration from [3], I employed a very naive product segment-based post-processing. The core principle is intuitive: we first segment the products into popular and non-popular groups (or more groups). Then, if a user preferred popular products in 70% of their profile, then our list of recommendation should also be 70% popular.
In particular, we will rearrange the base recommendation given by the recommender and match the user profile in each page. So, if each page can show 5 products and user expects 60% popular items, we should rearrange so that the each page contains 3 popular and 2 non-popular items whenever possible.
As a recommender easily outputs dozens of products for each user, it is easy to match the user profile for many pages. As naive as it is, it matches the user expectation while preserving the original ranking as much as possible, and also requires minimal effort.
Product Segment Bias
If we consider popularity as a dimension of product segmentation, then the same principle and techniques also apply to other dimensions like product discount rate, price point, seasonality, etc.
Exposure Bias
In our previous article, we mentioned that recommenders train on implicit and explicit user feedback. In particular, implicit negative user feedback essentially means there is no observed interaction between a user and a product. It’s easy to see how it is ambiguous. Unobserved interaction might mean uninteresting product to the user, but it can also mean that the user is simply not aware of the product [4]. This is a bias in the data itself.
How Can We Handle It?
If you have implemented event tracking on your website (let’s talk about it in a future article), it might help a little bit. You can limit negative user feedback to unclicked product impressions. However, even this poses a few issues: (1) number of negative user feedback is significantly lowered, and (2) even though a user sees a product without clicking it, it doesn’t necessarily mean “dislike” or “uninterest”.
Position Bias
As users are more likely to click on products at the top of the page, this creates a bias where the user feedback is dissociated with the actual relevance [6].
Previous Model Bias
This forms a strong positive feedback loop where the model feeds the user some recommendation, which are more likely to be clicked on, which in turn means the model training on such feedback in the future will amplify the bias. Combining with the effect from the other biases, this creates notorious issues like echo chambers and filter bubble [5], eventually isolating the users to a narrow segment of products.
Conclusion
We have talked about some biases in the data and in the model. However, some bias might even originate from the users themselves. For example, the user might click on some products if they see a high rating from other users, even though they might not be interested in the product itself [4].
Mitigating the performance impact from biases is an active research area in recommender systems. Learning to rank is an intricate class of study in its own right, and is often employed to handle the above biases. There is even recent attempt to use the bias to enhance recommendations [5]. As with all other machine learning projects, it is important to focus not only on the algorithm itself, but also to understand the bias and their consequences.
Reference
[1] Abdollahpouri, Himan, Robin Burke, and Bamshad Mobasher. “Controlling popularity bias in learning-to-rank recommendation.” Proceedings of the eleventh ACM conference on recommender systems. 2017.
[2] Abdollahpouri, Himan, Robin Burke, and Bamshad Mobasher. “Managing popularity bias in recommender systems with personalized re-ranking.” The thirty-second international flairs conference. 2019.
[3] Abdollahpouri, Himan, et al. “The unfairness of popularity bias in recommendation.” arXiv preprint arXiv:1907.13286 (2019).
[4] Chen, Jiawei, et al. “Bias and debias in recommender system: A survey and future directions.” arXiv preprint arXiv:2010.03240 (2020).
[5] Zhang, Yang, et al. “Causal intervention for leveraging popularity bias in recommendation.” Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021.
[6] Collins, Andrew, et al. “A study of position bias in digital library recommender systems.” arXiv preprint arXiv:1802.06565 (2018).
