Product Recommendations in the Newsletter — Crafting the MVP — Keep Calm and Run Your ML Projects the Lean Way — Part 3
Welcome to the third article of the series “Keep Calm and Run Your ML Projects the Lean Way”.
In the first article, we have defined the phases and introduced some tools used to break down a system into phases and visualize and decompose them into different dimensions.
The second article focused on the first phase — The Riskiest Assumption Test — RAT.
This has brought us to the next iteration — The Minimum Viable Product.
TLDR;
The second iteration through the Build-Measure-Learn Feedback Loop includes:
- Breaking down the solution using the Mercedes Decomposition framework
- System Design
- Measurements and Learnings
Mercedes Decomposition — MVP
Let’s rehearse the definition of the MVP, just in case you missed it from the first article:
Minimum Viable Product (MVP) - the first deployed version system in order to assess the viability of a set of hypotheses among a user base.
The figure below contains a set of hypotheses that we have made in this phase. The green sticker notes are used to highlight a hypothesis and red sticker notes are used for relevant questions.
As discussed in the first article, we add sticky notes along the 3 axes, but at this point, we are focusing on the second phase (the second circle) — the MVP phase.
Method — How do we do it?
In the RAT phase, we were simply sending the top-sellers from the customer’s category of interest.
In this phase, we are adding 3 procedures:
1. Filtering out products from the customers purchase history
2. Filtering out products from the customers newsletter recommendations history
3. Enforcing diversity per product category — if the customer only has one category in the purchase history, we need to make sure that the recommended products are comming from different subcategories.
Each of these procedures adds complexity to the solution, which is not ideal, but it is enough to achieve the goals of this phase.
Data — how do we compute this?
In the RAT phase, we started with using the Product Category Tree (associating products to categories and sub-categories) and the Customers Purchase History.
In this phase we will:
Start recording which products were recommeded to each customer.
We will simultaniously start consuming this data for for the formentioned filtering purposes.
User story — how do the products reach the inbox?
Most of the efforts are directed toward this dimension. In the RAT phase, it was clear that we needed to associate the customer_id to the product_ids we were offering and pass them to the CRM tool that takes care of sending the emails. In this phase we need to take care of a few more things:
1. We are automating the process — We need a scheduled job that runs before each newsletter send-out in order to generate fresh recommendations.
2. We need to migrate from a jupyter notebook to a peer-reviewed set of scripts, clean code (readable, reliable and maintainable) , logging, (idealy monitoring and alerting).
3. We need to persist recommendations to a database, which will enable us to track the recommendation history.
4. We need to implement an audience splitting meachanism for experimentation purposes.
System Architecture
Unlike the RAT phase, in this phase, we actually have a decent but still, simple architecture. We are migrating from a Jupyter Notebook and CSV files to a set of 2 Airflow jobs and a database.
The first Airflow job triggers the batch pipeline with the end goal to persist the customer_ids and the proudct_ids that have been selected for them.
The first step involves an audience split, where we divide the audience into two groups of equal size. This feature operates in two modes AA and AB. The first (AA) is used when there is no ongoing experiment and the second (AB) when there is a running experiment. In order to choose the mode, experiments are recorded in the database.
This step is followed by the main tasks — selecting candidate products that are mapped to the corresponding customer_id.
After we associate the candidate products with the customer, we need to filter out the products that have :
1. already been purchased by the customer and
2. already been recommended to the customer (in the last month).
As the list of candidate products is often larger than the number of products that are being sent to the customer, we need to rank them and select the top 4 products.
Finally, if the customer is left with fewer than 4 products to be persisted, the fallback strategy is applied in order to ensure that they receive the needed amount of products.
The recommendations are then persisted into a relational database, along with the date when they were generated, for the purpose of tracking the recommendation history.
The second Airflow job runs consequently to the successful completion of the first one in order to upload the latest recommendations to the CRM tool, which takes care of sending the emails to the selected customers.
We are monitoring the number of unique recommended products, number of unique products per category, the number of customers that are receiving fallback recommendations, etc.
At this stage, the system still doesn't have an understanding of the availability of products (are they low on stock or even unavailable)
The Measure & Learn Part
Too many rules are needed to make this system work in the long run.
The products will undoubtedly become repetitive for most of the users, which will lead to many of them receiving fallback recommendations after 5–6 weeks (because we explicitly remove the previously purchased or recommended products).
We need a method that will be able to come up with a larger number of relevant candidate products for each customer.
We need ML 🥳
Our recommendations are too narrow (often limited to a single category per customer) unless we explicitly enforce diversification. We need to recommend products of different purposes (such products are often bought together, or bought over time by a single customer).
We need a method that will be able to come up with relevant products that come from categories that the customer has not explicitly shown interest for.
We need ML 🥳
Conclusion
In this third installment of our series, “Keep Calm and Run Your ML Projects the Lean Way,” we’ve dived into the crucial phase of crafting the Minimum Viable Product (MVP) for our product recommendation system. The MVP is the first deployed version of the system designed to assess the viability of our hypotheses within a user base. During this phase, we focused on refining our approach from the first phase and making strides in various dimensions.
Despite the progress, this phase also revealed certain limitations and challenges. The need for too many rules to sustain the system, product repetitiveness for users, and limited diversification in recommendations highlight the necessity for machine learning to enhance our recommendation capabilities.
In the upcoming fourth installment, we will explore deploying a scalable solution to address these challenges and continue our journey of running Machine Learning projects the lean way. Stay tuned for more insights and practical strategies in our ongoing series.
Coming soon: Personalizing Product Recommendations for the Newsletter — Deploying a Scalable Solution — Keep Calm And Run Your ML Projects The Lean Way — Part 4
Hit the subscribe to get notified about upcoming articles! ❤️
