Recommender Systems: The Information Enhancement Trick

Hello and welcome to another article on my attempt to tame the seas of Recommender Systems and fight the Kraken (or product-you decide how to name it!)

In my two previous articles, you saw my methodology on how to master working with products, annihilating the monster that made things very difficult.

You see, the more the data (the products and customers of the shops in our database, in our case), the larger and more complex the calculations. Large and complex calculations take up a huge amount of time. I had a proper Lernaean Hydra to tackle!

I needed to take a lot of time to solve this problem, but I had another task at hand as well: Like any good hero would do, I needed to help out smaller shops with interaction data, as theirs were simply not enough.

In this article though, I will…

(Drumroll, please!)

…Investigate the impact my quest had on our model.

Oh, and if you’re feeling lost, here’s your map.

But let me remind you what Recommendation Systems are, first:

A recommendation system is a system that aims to predict the preferences of a user when it comes to an item.

It uses either the data from a user’s past interactions with an item (views, purchases, ratings, etc.), in order to make recommendations of similar products to the user, or some key components of the item the user has interacted with, which will result in recommending items with similar features or use.

But I know you’re not up for long narratives, this is action and we need to get straight to the point. So, without further ado:

Product clustering and entity recognition steered my wheel in a different direction

In order to avoid more traps, I used the same structure as my first model, but with a twist: I applied the principles of Product clustering and Named Entity Recognition to the whole process.

Let me tell you, it did make things somewhat safer.

For those of you who don’t remember, Product Clustering is the process through which I managed to put all same or similar products of different brands under the same cluster.

Named Entity Recognition is the process through which we seek and identify information units in unstructured text and classify them into predefined categories.

Pretty handy, if I say so myself!

Now, before merging the interactions of the customers with the products, I went through with product clustering.

This created groups (clusters) of products.

The next step was to assign the interactions of the customer to the cluster that the products they interacted with, belong.

This lead all the information that I already had for each product to be enhanced. I used the interaction data and shared it across all the products in the cluster.

Let me break it down a bit for you:

John bought a Nike t-shirt, Jane bought an Adidas t-shirt and Peter bought Adidas shoes.

The t-shirts ended up in the same cluster. However, the shoes didn’t.

The interactions were assigned like this:

But there was also another thing to keep in mind: A cluster contained products from only one gender.

Therefore, it wasn’t possible for someone to select a product that belonged to another gender, with the exception of the neutral (unisex or otherwise not specified) products.

Model Workflow

The woes of selecting a product (and how I solved the problem)

Selecting a product to recommend from a cluster is not all fun and games. These little buggers can get a bit tricky sometimes.

So, I had another problem in my hands that I just had to solve, in order for my endeavors to really matter.

I tried to find a way to score each individual product in the cluster according to popularity. But there was a parameter I didn’t keep in mind: This made me end up selecting the exact same product from the cluster, over and over again.

The problem with that was the fact that the highest score was assigned to only one product each and every time and there was no other kind of data to salvage the rest.

Now, as if that was not enough, this scoring method enabled another monster to appear:

A product for men ended up in a cluster with women’s products.

I had no idea how to get out of that one and the only logical solution at the time was to randomly select each individual product from the cluster.

It ended up being a numbers game

As evident, I was at a small loss at that point. However, there was no solution but to push through-even if this meant that I would crash and burn in the process.

I used the same metrics that I’d used for the first model I’d built, but there was a small problem stalling the whole process: the clusters.

At that stage, I had product clusters instead of individual products. The clusters are comprised of the same or very similar products.

This left me with no choice but to measure the performance of my “ship” in two different levels: The cluster and the product level.

The results are pretty clear here. My first model was incomplete, but by adding product clustering and then the named entity recognition, I managed to improve my original model’s individual components and push through.

• Its accuracy was increased, from 2,3% to 7,58%
• Its precision was increased, from 0,51% to 1,45%
• Its recall rate was increased, from 1,8% to 2,06%

So far, so good and this trip was branded as a successful one!

Or so I thought. But I was quick to judge…

Numbers do lie (if you count wrong)

Yes, the size of the matrix was decreased when I applied product clustering. The computation time was decreased as well (Hooray!). But this doesn’t mean I managed to bring it down to a working size, it was still too large to manage.

The other issue I faced, was the fact that the calculations were too complex for my plan to work properly.

But it didn’t stop there.

My calculations were way off on a thing or two (stay tuned to find out how I managed to fix all that!)

As I observed the products that model recommends and compared them to the product interaction information that I fed to my model, I realized that something was more than a bit off.

There was no distinction between the interactions with the products (views, add to cart, purchases, you name it!) and the fact that the customers were different individuals.

This was simply not a part of the equation at all.

To me, it sounded like a thunderstorm was approaching. I had no option but to find a solution.

And I did find it. But I won’t tell you how just yet. Tune in for my next adventure. More monsters, complex mechanics and heroic quests await!