How to match Ketchup and Pasta with AI?
Complementary recommendation algorithms are like magic helpers that suggest things that go well together, just like peanut butter and jelly or movies and popcorn. These algorithms work by analyzing patterns in what people choose or buy. Imagine you’re ordering pizza, and the recommendation system suggests garlic bread or a cold drink to go with it. It’s not just about what’s popular or similar; it’s about what complements your main choice. The algorithm looks for items that enhance or complete the experience, making it more enjoyable or useful for you.
For example, think about pasta and ketchup. While pasta is great on its own, adding ketchup can make it even tastier for some people. A complementary recommendation algorithm might notice this pattern and suggest ketchup when someone orders pasta. Similarly, if you’re buying a smartphone, the algorithm might suggest a compatible case or headphones to go with it. By understanding what pairs well together, these algorithms help make our choices more satisfying and convenient.
From First Interaction to Last Embedding
Our complementary algorithm builds upon the foundation of the Shopper algorithm.
It delves into predicting user intent based on their purchases and considers factors like price changes and user interactions. It answers questions like, “What happens if I increase or decrease the price of this item?”. This approach allows us to understand not just what users buy, but why they buy it, and how they might react to changes in price or other factors. As part of this process, the algorithm generates two key types of embeddings: Item embeddings and Shared Item embeddings. These embeddings serve as unique representations of items and their relationships within the shopping ecosystem.
For instance, consider a scenario where a user frequently purchases both a smartphone and a protective case. The algorithm would create embeddings for both the smartphone and the case, capturing their individual characteristics. Additionally, it would generate a Shared Item embedding that represents the relationship between the smartphone and the case, recognizing that they are often bought together. This approach enables the algorithm to identify complementary items that enhance the user experience, such as suggesting a screen protector when someone buys a smartphone. By leveraging these embeddings, our algorithm provides personalized recommendations that are not only relevant but also seamlessly complement users’ existing choices.
All Pipeline
The journey of the data during the Complementary calculation is exactly as above. Of course, some small side functionalities are not added to make this graph even larger, but obviously, other tools are used in the background for logging or orchestration. Today, let’s talk about what is done for complementary calculation, which is the purpose of this article.
Dataset Preparation
To power our complementary recommendation algorithm, we begin by collecting a rich dataset of user interactions with various items. Every click, purchase, and view is logged as an event, providing valuable insights into user behavior. These interactions are then aggregated and organized into a tabular format, forming the foundation of our dataset. However, not all interactions are created equal, and some outliers can distort our analysis. For instance, instances of excessive activity within a short timeframe or suspicious bot clicks can skew our understanding of user preferences. To ensure the integrity of our dataset, we meticulously filter out these outliers. For example, we might identify and remove instances where a user has viewed an unusually high number of items in a single day, suggesting potential bot activity or data anomalies. By eliminating such undesired values and grouping user interactions into coherent sessions, we can focus on meaningful patterns and deliver more accurate recommendations to our users.
At the end of this process, we now save this data in S3 to be used in the train in the future. Here we use our own ML Platform, Delphi. We write the number of rows related to the dataset we prepared in the dataset table in Delphi, along with some information about the partner, etc. In the future, when the automatic train is running, it will come here and take the last saved dataset guid and use it to train.
By failing to prepare, you are preparing to fail.
-Benjamin Franklin
Training is Nothing Will is Everything
The title, which is a quote from the movie Batman Begins, may be valid in the movie, but unfortunately, I will not be able to participate in Ras Al Ghul for the Complementary calculation.
Once we have meticulously prepared and refined our dataset, it’s time to harness its power and put our algorithms to work. Leveraging the data stored in S3, which has been meticulously organized and cleaned during the dataset preparation stage, we initiate the next phase of our process. Here, we turn to Amazon SageMaker for the Training Job, a pivotal step in refining our recommendation algorithm.
To expedite the training process and unlock greater computational power, we capitalize on SageMaker’s support for GPU-accelerated instances, particularly the g-type instances. These instances turbocharge our training efforts, enabling us to iterate faster and fine-tune our algorithms with unprecedented efficiency.
Before triggering the TrainingJob, we ensure that the necessary code is readily available and primed for execution. This involves pushing the requisite code into Amazon Elastic Container Registry (ECR), a crucial component of our deployment pipeline. We offer two pathways for code deployment: one involves seamless integration with GitHub, where changes trigger automatic actions, streamlining the deployment process. Alternatively, for more localized control, we provide the flexibility to push code locally. Regardless of the pathway chosen, the end goal remains the same: to encapsulate our algorithm within a Docker image residing in ECR, ready to be deployed seamlessly by SageMaker. This streamlined workflow ensures that our algorithms are always up-to-date and ready to tackle the complex challenges of recommendation generation.
In our quest to refine the complementary recommendation algorithm, embeddings play a pivotal role. These embeddings serve as compact representations of items and user preferences, enabling us to distill complex interactions into manageable dimensions. While numerous embeddings contribute to the algorithm’s efficacy, two stand out as particularly influential: item-item embeddings and basket average embeddings. The embeddings trained here are created based on the equation in [1]. For more detailed information, you can see the link in the Further Reading section.
Item-item embeddings capture the intricate relationships between different items within our catalog. By analyzing historical user interactions, we generate embeddings that encapsulate the similarities and complementary nature of various items. These embeddings form the backbone of our algorithm, guiding the recommendation process towards items that seamlessly complement each other, such as suggesting a smartphone case alongside a new phone purchase.
Finally, we put these item-item and item latent embeddings somewhere in S3 so that we can pull them directly from there for the inference stage later.
Inference Step
During the inference phase, our algorithm conducts matrix multiplication involving two matrices, potentially expanding to massive sizes. On one side, we have the items themselves, while on the other side, we incorporate shared or additional items within the basket. Presently, the basket typically contains only one item, although this number may vary in the future. This matrix multiplication process allows us to generate predictions based on the intricate relationships between items, ensuring that our recommendations remain tailored and relevant to each user’s preferences and context.
Example Recommendations
It would be good to show one example in the blog as an example. Here, when we look at the knife, we see that the partner suggests a cutting board.
Further Reading: Humboldt Complementary Calculation
[1] Ruiz, F. J., Athey, S., & Blei, D. M. (2020). SHOPPER. The Annals of Applied Statistics, 14(1), 1–27. ISO 690
Hope you found the article useful. If you have any questions, feel free to ask away in the comments. You can also follow our Insider Engineering Blog for more articles where we detail our engineering processes. Here is a starter for you to check out! How Our Auto ML Platform Handles Feature Selection.
