The current state of recommender systems
Or, why do I still get prompted every time to watch that awful Adam Sandler movie?
Prelude
Let’s be honest — AI is not only a thing. AI is huge. And we are not just talking about the amount of recent media coverage, but also its potential to change the world as we know it. Experts are mostly no longer debating about the “if”, but the “when”.
As word started to spread more pervasively in media about breakthroughs of Big Data fuelled Neural Networks, it sparked an interest that led me down the rabbit hole. From some light reading, I headed on to Udacity Nanodegrees in Deep Learning and Artificial Intelligence on a quest to understand the topic better.
A year and a half later, news stories appear to still be created at a rapid rate and scientific papers on the topic are published in the dozens each week.
During Infiniteconf 2018 recently in London, recommender systems received dedicated talks as well as honorable nods in almost any presentation covering Big Data or AI in general.
Sure, some of the following might be orchestrated by my personal filter bubble, but it sure is worth being in-the-know about what is going on.
What is a recommender system?
Let’s look at one of the many things that are getting huge benefits from exploding data volume and available computational power. Recommender or recommendation systems are probably as old as trade itself. The gist of it is that based on things you like or have enjoyed or purchased in the past, somebody or something suggests a similar thing that you might also find of interest.
I see you like wearing sheep wool loafers, how about owning the whole sheep? For a price of course.
- probably somebody in Mesopotamia, 10000 B.C.
Not sure what to watch on Netflix? Google searching how parts of the world became a political mess in the recent years? Need another Weber Master-Touch BBQ, similar to the one you bought last week on Amazon? Your phone warning you when you mistyped “duck”? You guessed it, recommender systems to the rescue.
Due to the plethora let loose on unsuspecting customers worldwide it should be clear by now what they do. But how do they function?
A brief history
Most of the early systems were all come up with by manual (guess)work. Kind of: “Sir, would you like a key cut while waiting for your shoes to be cobbled?”. This has been being replaced by slightly more automated systems since the advent of online shopping. Products in certain categories were bundled together, or purchases often made in combination flagged to customers buying part of a typical combo. Want some beer with those snacks? Of course, I do!
Improvements were mainly made by swaths of data engineers or other people across the company who came up with new categories or pairings to try.
Recently, whoever has some cash to splash has pumped their systems to the next level by gathering reams of user-created feedback and data, like star-ratings, feeding it into a neural network and waiting for it to be digested. The outcome - a better, cheaper, scalable sales machine, tailored to individual buyers.
The current state
There are many paths to similar results, but here is a short description of how Netflix’s algorithm, as an example of these systems, is currently functioning.
It all starts off with huge amounts of data.
On one hand, this data comes from users in the form of explicit ratings or likes of certain movies or shows they watched. It can also come from implicit sources, like binge-watching an entire series over short periods of time. What people say they are watching (documentaries, political dramas,…) often does not match what they are actually consuming (Love Island,…).
On the other hand, Netflix is also dissecting the content they are offering to users. This can range from simply tagging the genre of a movie, to a far more nuanced analysis of scenes, characters or styles of their video material. The trick is then, to figure out which types of users are drawn to which types of content, for which reasons.
This is where machine learning comes into play. Neural networks have the task to assign weights to hundreds of different input variables and decide which factors have the biggest role in deciding which material any given user might be interested in next. And they do a pretty good job at it, given enough input and computational resources.
While this system still has its flaws, around 80% of Netflix users are now using the company’s recommendations to discover what to binge on next. That is huge, given the over 100 Million unique users the service has.
If you would like to find some more in-depth explanation of different examples from various implementations, have a look at this interesting talk by Alex Smola.
In the not so distant future
State-of-the-art machine learning is evolving rapidly. Huge investment from Academia and Industry alike is creating promising novel systems and ideas.
One approach recently making a lot of headlines is called Reinforcement Learning. In simple terms, it is the AI equivalent of Laissez-faire parenting. Instead of telling your algorithm exactly how all parameters look, how they are related and giving example desired outcomes for each input, the software is given a goal and access to input data. The rest should be figured out on its own, usually by lots of trial and error.
Oliver Gindele, from DataTonic, held a very interesting talk on the topic at Infiniteconf. He also believes that in the near future, the underlying models will become increasingly complex and less rigid. They might draw from different moving parts as needed to improve recommendation accuracy.
Epilogue
Recommender systems are sometimes spot on, and at other occasions failing miserably. (Deep) Neural Networks are doing a fine job and are driving current improvements, but there is still room for more accurate systems. Rest assured, a lot of budget is flowing into research to make us spend more money while enjoying every bit of our purchases.
If you would like to know more about the recommender systems or AI in general, I can highly recommend the Udacity courses and Nanodegrees on the topic! They also introduced a Data Scientist Nanodegree now that covers recommender systems as one of their projects for students to build.
