Discovering the Closest Alternative Products with Substitute Recommendations
In the realm of recommendation systems, the Substitute Recommendation algorithm emerges as a pivotal tool, providing users with alternatives that seamlessly align with their preferences. At its core, this algorithm is designed to identify substitute products, offering users a curated selection that serves as potential replacements for items they have engaged with.
The substitute products recommended by the algorithm are informed by a range of factors, including product name similarity, as well as product visit and purchase statistics, which are key in determining suitable alternatives.
Taking these factors into account, the algorithm ensures that the recommended substitutes not only perform similarly to the user’s initial choice but also align with broader preferences commonly observed. By analyzing a comprehensive network of data points, the system discerns general patterns in user interactions to offer relevant suggestions. The algorithm essentially focuses on identifying trends across product usage and engagement. This approach allows our substitute recommendation algorithm to present a dependable array of alternatives, aiming to enhance overall user satisfaction by broadening the spectrum of choices.
General Flow
The diagram illustrates that the core of the Substitute Recommendation algorithm is an EMR-based Spark Job designed to fetch and process both product catalog and user event data from S3 storage. Once retrieved, this data undergoes further analysis to generate substitute recommendations, which are then stored in ElasticSearch for access and retrieval.
The delivery of generated recommendations occurs via the Recommendation API. Upon users visiting a product page covered by a Substitute Recommendation campaign, a request is dispatched to the API’s substitute recommendation endpoint. This request comprises the product’s ID and the parameters and filters defined at the campaign’s setup. It then retrieves the substitute recommendations for the specified product from ElasticSearch, showcasing them on the product page for the user.
The initiation phase of the substitute recommendation algorithm involves extracting product catalog and user event data from the Iceberg tables stored in S3. Within the Product Catalog data, the product names undergo a sophisticated transformation using the Word2Vec class in Spark ML library to generate intricate word embeddings. Word2Vec in the Spark ML library utilizes a neural network-based approach to learn distributed representations of words. The model is trained on a vast collection of product names, learning to predict the context (surrounding words) of a target product name or vice versa. This training process results in a set of word vectors, or embeddings, where each word is represented by a high-dimensional vector.
These word embeddings exhibit interesting properties. Words with similar meanings or contexts tend to have similar vectors, making them close to each other in the vector space. The geometric relationships between these vectors can reflect semantic relationships between words.
This intricate process contributes to the creation of rich and nuanced representations of the product names, enhancing the algorithm’s ability to discern subtle semantic relationships and patterns within the data.
Product Name Similarity
For the purpose of calculating similarities between product names, it is necessary to first generate word embeddings for these names. The Word2Vec class within the Spark ML library, utilizing the Skip-gram algorithm, has been employed for this task.
import org.apache.spark.ml.feature.Word2Vec
val word2Vec = new Word2Vec()
.setInputCol(inputColumnName)
.setOutputCol(outputColumnName)
.setVectorSize(vectorSize)
.setMinCount(0)
val model = word2Vec.fit(dataset)
val transformedDataset = model.transform(dataset)The Skip-gram algorithm is a popular method used for creating word embeddings. The primary goal of this algorithm is to learn the dense vector representation (embedding) of a word based on its context. The algorithm typically takes a large corpus of text (in our case product names). Each training example consists of a “target” word and the “context” words found in the vicinity of this target word. The context is usually defined as the words within a certain window size around the target word (for example, two words to the left and right of the target word).
The Skip-gram model uses randomly initialized vectors (embeddings) for each word. The model tries to predict the vectors of the words in the context using the vector of the target word. This process is typically carried out using a neural network architecture, during training, the word vectors are updated to reduce the difference between the model’s predictions and the actual context words. This is generally done using the softmax function and the backpropagation algorithm. As a result, a dense vector is obtained for each word. These vectors reflect the semantic similarities and relationships of words. For example, words like “king” and “queen”, which are used in similar contexts, will be close to each other in the vector space.
We can create hypothetical vectors for each product name to calculate cosine similarity. These vectors are simplified representations for illustration purposes.
The cosine similarity scores for each substitute product in relation to the “Deluxe Coffee Maker” are as follows:
View to View Statistics
Understanding the significance of “View-to-View” patterns in online shopping sessions is crucial for enhancing recommendation systems, particularly in the context of identifying substitute products. This approach leverages the frequency and sequence of product views within the same session to infer potential substitutability among items. When customers browse through various products in a single session, they often compare features, prices, and other attributes to make informed purchasing decisions. By analyzing these browsing patterns, we can deduce which products are being considered as alternatives to one another.
For instance, if a significant number of users view Product A and then Product B within a short timeframe during the same session, it may indicate that these products serve similar needs or preferences for those users. This pattern becomes more pronounced when observed across numerous sessions, suggesting a stronger substitute relationship between the two products. The “View-to-View” score can be quantified by tracking these occurrences and applying statistical models to gauge the strength of the substitutability.
The number of times products are visited within the same session is calculated as an example and scaled between 0–1 using Min Max scaling.
View Then Purchase Statistics
After covering product name similarity and session-based browsing patterns, let’s turn our attention to a more behavior-oriented metric: the View Then Purchase score. This particular statistic sheds light on how customer preferences evolve from initial interest to the final purchase decision. Imagine a scenario where a customer lands on one product page, let’s say a coffee maker, spends some time there, and then, without making a purchase, navigates to another coffee maker and buys it. This sequence — viewing one product but buying another — sheds light on a potential substitute relationship between the two products.
Why does this matter? Well, it’s like reading between the lines of customer choices. If several customers are consistently viewing Product A but ending up purchasing Product B, it might suggest that while both products catch the eye, there’s something about Product B that clinches the deal. It could be anything from a better price, more appealing features, or even just the way Product B is presented.
This pattern is useful for understanding what customers might consider a ‘Plan B’ for their desired product. It’s not just about what products are viewed together in one session; it’s about understanding the final choice. This insight can help us fine-tune our recommendations, ensuring we’re not just throwing random suggestions at our users but providing them with thoughtfully selected alternatives that align with their initial interest.
The view then purchase counts for products are calculated as shown in the example below and scaled between 0 and 1 using Min Max scaling.
Substitute Score Calculation
The algorithm computes the substitute scores by averaging the metrics derived from product name similarities, view patterns, and purchase behaviors. This holistic approach ensures that the recommended substitutes are not only semantically similar but also align with user preferences and behaviors observed across multiple shopping sessions. This approach also enables the generation of substitute recommendations for newly added partners in the system that may not have sufficient user activity data, by leveraging the semantic similarities of product names.
The substitute scores for products, for which name similarities, view-to-view scores, and view-then-purchase scores have been calculated, are determined by taking the average of these three values, as shown below.
Based on these substitute scores, the substitute recommendations for the Deluxe Coffee Maker product are ranked as follows:
1- Premium Coffee Machine
2- Ultimate Espresso Device
3- Basic Brew Maker
In essence, our Substitute Recommendation algorithm works by examining product names, how often products are viewed and bought, and then crunching these numbers to suggest similar items. It’s a straightforward process designed to help users find alternative product.
Substitute Recommendation Example
To observe the real-world performance of the Substitute Recommendation algorithm, an example has been provided from a campaign on an e-commerce partner’s website.
In the image above, the main product is a gaming laptop featuring an Intel Core i7 12700H processor, 16GB of memory, a 1TB SSD, and an RTX 3070Ti GPU. The substitute recommendations listed are also gaming laptops with specifications aimed at a similar target audience. They each feature variations of Intel Core i7 processors, comparable amounts of memory, SSD storage, and RTX GPUs, though the GPU models vary slightly, offering a range of performance options.
The accuracy and relevance of the substitute recommendations seem quite appropriate, as all the alternatives maintain the gaming laptop theme and share core specifications that matter most to gamers: high-speed processors, sufficient RAM, fast SSDs for storage, and powerful graphics cards. These recommendations would be helpful to a consumer who may be looking for options similar to the main product but with slight differences in features or price points.
I hope this article has provided you with valuable insights. If you have any further questions or comments, please feel free to express them in the comments section or reach out to me on LinkedIn. To gain more insights into the inner workings and performance measurements of our recommendation system, you can read this extensive article that covers these subjects in detail.
