Beyond Recommendation Engines
Reusable Multipurpose Recipe Embeddings using Graph Neural Networks
By Jessica Tyler, Bruno Dupont, and Luca Fiaschi
At HelloFresh, giving the customer an increasing array of recipe choices has played a crucial role in our success. Our current menu in the US features 25 new recipes and 50+ add-on items each week — in other markets even more — allowing us to offer thousands of different options to millions of customers.
This wealth of data represents a significant opportunity for our business to meet customer needs. Understanding customer tastes and preferences can help:
- Engage customers with their favorite food items through email and other marketing communication, increasing the order rate
- Improve product discovery, encouraging customers to place more items in their basket at checkout (increasing average order value)
- Guide our culinary teams as they perfect menus, matching them to our customers’ preferences
In HelloFresh, our Data Science and Analytics departments have dedicated many teams to optimize every aspect of our business, creating a range of Data Science Products using data from customer choices.
For instance, the US Marketing Analytics team develops a content-based recommendation engine based on recipe similarity scores to increase user engagement.
Meanwhile, our Culinary Analytics team uses cluster analysis of recipe and recipe similarity to understand white spots in our menu offering.
What do these applications have in common? Before applying techniques from the traditional data science toolbox — classification, regression, and clustering models, the recipe data must be transformed into a vector of numerical values called embeddings.
Embeddings must preserve the semantic relationships between the recipes to be helpful in downstream tasks such as recommendations, forecasting, and culinary development. For instance, we can use cosine similarity scores to compare two recipe embeddings. A cosine similarity score of 1 means that the recipes overlap entirely in our vector space therefore they are semantically similar. In contrast, a score of 0 means that the recipes are different from each other. See below a hypothetical example with three recipes in a dimension-reduced representation of our vector space below:
The objective of our team is to provide a starting point for other Analytics teams and business stakeholders to make more effective use of our recipe data in advanced AI applications. We do that by delivering and making available standardized recipe embeddings to our analysts.
This article describes how our team constructs those recipe embeddings.
Building the Recipe Knowledge Graph
Graph Neural Networks [1] are a state-of-the-art class of machine learning models for embedding recipe data because they can well represent the semantics of relationships between the many factors that make a recipe unique.
At a high level, there are three stages to building such a model:
- Define what goes into the Graph
- Train a skip-gram neural network model to get node embeddings
- Store embeddings in downstream feature store for data science applications
The first element of this was to create a Knowledge Graph of the recipe data. How does this play out for our use case? The image below is a graph representing two recipes HelloFresh has featured on the menu in the past. The recipes are represented as red nodes, with edges connecting to the ingredients (tan nodes) they contain and other recipe-level traits (meal type in light blue, total calories category in pink, etc.) If a recipe has a particular trait, it will have an edge that connects from its node to the node representing the trait.
For example, we can see that both the “Creamy Dill Chicken” and “Spiced Chicken Cutlet” recipes have edges connecting them to the “low total calories” node: both are low-calorie recipes. We can see, however, that even though the first contains mustard, the second does not.
We call this the Recipe Knowledge Graph because it encodes everything we know about our recipes, making it easy for stakeholders to explore and understand the data. But how do we use it to determine which recipes are similar? Here is where a skip-gram graph neural network (Node2Vec) model comes into play. This model allows us to learn from the relationships in a graph and create embeddings for every node.
Training Recipe Embeddings
We use Node2Vec, a simple extension of the famous Word2Vec algorithm, on the Knowledge Graph defined above. Node2Vec acts into two steps:
- Sample random walks (path of nodes) that are representative of the larger Graph structure
- Embed each node in the sampled data using the Skip Ngram Neural Network Model
How does that look with our data? The image below shows some random walks of length six extracted from our knowledge graph.
In the above example, we highlighted a window around some of the nodes (cyan squares), that includes two nodes preceding and following (i.e., nodes that fall inside the black rectangles.). The skip-gram model is a shallow neural network (consisting of only one hidden layer) that predicts the likelihood of a node being found in the same neighborhood (i.e., in the same window) as any other node in the graph. To get a sense of what this means for our use case, refer to the illustration below.
The recipe embeddings (blue neurons in the figure above) are vector representations of all nodes in the graph corresponding to the hidden layer of the neural network — and they encode semantic similarity across nodes that often co-occur together.
Below is the Knowledge Graph representation of two recipes with a high cosine similarity score:
The recipes (red nodes) are “Roasted Pork with Balsamic-Fig Sauce, Green Beans, and Rosemary-Roasted Potatoes” (the recipe on the left) and “Cranberry Dijon Pork Chops with Roasted Root Veggies and Brussels Sprouts” (the recipe on the right.) We can see that these recipes have many shared traits, such as cuisine, total prep and cooking time, and protein category.
Although not all ingredients (tan nodes) in our recipes are shared, the non-common ingredients are also not central to the recipes. For example, only the vegetable sides and some sauce ingredients (mustard, shallot) differ. Further, while the meal types (light blue nodes) are technically not the same — they aren’t all that different — the “Cranberry Dijon Pork Chops” would just need a starch (e.g. potatoes) to fall under the category “Protein Starch Veg” meal-type instead of “Protein Veg.”
Recipe Embeddings within Our Data Science Strategy
Our recipe embeddings are a stepping-stone within our strategy to provide analytics and data science teams with building blocks to accelerate building AI products. We do so to democratize access to high-quality machine learning features that can help our organization accelerate the completion of data science and analytics projects.
Building embeddings for the first time can be a daunting task because of the difficulty in fine-tuning neural networks and because of the size of the dataset. However, since this embedding can be precomputed at multiple resolutions and stored in our data lake, they can be re-used in various downstream data science products and allow the data scientist to focus on innovations and unlock business value.
In addition to this, we are currently working on a corporate-wide Feature Store, collecting hundreds of features to more efficiently produce and scale machine learning and AI applications. More about that in our future blog posts!
[1] A Gentle Introduction to Graph Neural Networks, Distill 2021
