Psychology of “Frequent Pattern Mining” & Why They Succeed As a Recommendation System?
If you have some experience in Recommendation system and if you have thought
“I have been asked to build a recommendation system. I really want this sophisticated Deep Reinforcement Learning Model to Outperform the baselines but I having hard time understanding the Cost to Benefit of these new methods.”
then,
Welcome to the club of Deep Learning / AI evangelists , those have a conviction that complex methods might beat classical ML models. After all we crave for complexity in every aspect of our life! ML is no such exception.
It is true that Data Scientists who have specialised into these domains look for simplicity in solution as it is easy to communicate to business stake holders. Many a times, I have seen them work too! At least we knew why they won’t!
But with the advent of DeepLearning and the exponential increase in Online DL Courses , many people breaking into data science is convinced about the Super power of these neuron -inspired models. When given a problem, they directly go to their python IDE and type. “ import tensorflow as tf ”.
With latest methods being marketed over news like “ X models outperforms human, hence used by Y company” , we are motivated to try them out. But we also overlook the classic methods and baselines.
I do understand that our time is limited for a project. This may not be useful for a Computer Vision/ Natural language Processing problems. But for Recommendation Systems, might be!
Just saying that I am not against NN or RL (In fact, much of my work now is in Deep RL).Yes, they are very powerful. In fact, I have seen the representation power of Neural Networks, even for some tabular data (really stunning!). Also the RL’s adaptability in case of cold-start situations.
But when it comes to building recommendation systems and especially from scratch, I always had a strong conviction on efficiency of simpler sub-domains like Frequent Pattern Mining . If you are not familiar with the above term, you would have seen this section “Frequently Bought Together” at least once your online life.
It all happened one day, when I was reading “Atomic Habits” by James Clear. Yes, you heard that right. I was thinking about ML models while reading a self-help book on habits!
An interesting chapter in the book talks about building habits through a technique called “Habit Stacking”. It talks about how stacking your habits one by one increases the overall success of performing them.
The author also mentions about Diderot Effect . Inspired from the essay of a person named Diderot, it talks about how his new dressing gown made him to buy a lot of unnecessary furniture and goods, eventually putting him into debt. Although it is a sad story about irrational impulse buyer, we may have to remember that a significant portion of purchase decisions we make in today’s world happens at a blink. Of course, let’s exclude monks and buddhas!
So while reading about “Habit Stacking” & “Diderot Effect”, one familiar concept came to my mind. Yes, it is “Frequently Bought Together” way of recommendation. We are psychologically habituated to fix to the items that are bought together (in sequence!). Thus buying one item automatically triggers our impulse to look for the next item that has bought frequently together because they get stacked into our purchase routine.
This is one of the main reason why “Frequently Bought Together” recommendations , however simple , every time…….WORKS! especially for domains like e-commerce and retail, where significant number of items are purchased in a single session.
Trying to understand the co-purchase pattern of consumers, helps us to understand the psychology behind their preferences. Hence we can recommend better items (at better profits too!) but without spamming them with junk. I have seen companies, mostly stick to just one frequently bought item (may be because of average transaction length could be 2!). But mining a longer chain of these Item Stacks(yeah , I coined it!) revealed interesting insights in one of the uses cases for my client.
Classical methods like Apriori, Frequent Pattern Growth, PrefixSpan are used to mine these frequent and sequential patterns. You can check out this interesting framework called SPMF(Sequential Pattern Mining Framework). Also, we can go for Advanced Sequence Prediction Frameworks like Compact Prediction Trees, HMM, RNNS, Transformers.
If you are building recommendation system for your client, you may consider sequence prediction. But before that, there should be some design considerations as to:
- What kind of products have been sold ?
- How many products (on an average) has been bought ?
- Does mining these item stacks is useful for your other recommendation use cases like up-selling , cross-selling etc?
It is an interesting point of psychology meets ML. In the End, all our ML models are trying to capture the intents /psyche/choice of our consumers. By trying to connect these two worlds together, I am sure that we will be empowered more as Data Scientists / Story Tellers. What do you think ?
