Getting 🍨 Ice Cream 🍦 Recommendations at Scale with Gemini, Embeddings, and Vector Search
“I wonder if there’s another cookie dough ice cream as good as the one I had yesterday!” — me, more often than I’d like to admit 😀
If you’re a food aficionado, enjoying the current dish you have is often only the beginning: your mind will quickly pivot to how you can get something else that might be as good (or better!) than what you just had. Flip that to the corporate side, and an important part of food product discovery, marketing, and placement is knowing what other products are similar to ones users already enjoy, to help make effective recommendations.
There’s a long history of quantitative ways to make recommendations at scale, and Google Cloud has multiple documentation pages that cover different techniques. In this post, we’ll see how generative AI opens up even more possibilities for making recommendations among products — using long-form text data to build our comparisons, at scale, without leaving our data environment!
More Ice Cream!
In a previous post, we showed how to use Gemini, Google’s multimodal foundation model, to generate new text assets for each item in this well-curated Kaggle dataset of 240+ ice cream products — including shorter product descriptions, alt text, and review summaries. We encourage you to explore those results in this interactive Looker Studio dashboard.
Now, we’ll build off that expanded dataset to create recommendations amongst those 240+ products by leveraging text embeddings and vector search — all within BigQuery, Google Cloud’s fully managed AI-ready data analytics platform. That’s right: we’re actually going to find the next best ice creams to try to satisfy my cookie dough cravings! …And help retailers know what items to recommend to customers with varying ice cream palettes.
If you want to skip ahead to the results, try out this Looker Studio dashboard, which allows you to pick a single ice cream product up top and get a ranked list of comparable products in the table below. This also includes the data used to create comparisons across all products so you can see where the similarities/differences come from.
To learn how to create your own GenAI-powered recommendation system for products like these, read on!
Getting Started
We’ll leverage the setup and results from our previous post, which uses Cloud Storage and various aspects of BigQuery to generate more info on each product (see code on GitHub). The results, including Gemini-generated short product descriptions and review summaries, are stored in a table called `ice_cream_products.products_results`, with the following schema:
It’s relatively straightforward to take this table and compare products by average rating, match those within the same brand, or maybe even do some string matching on product names. But what if we want to go further and compare longer product descriptions, reviews, and images to find similarities?
Enter embeddings: numerical representations of text, images, or videos that capture semantic relationships between inputs. They are one of the most versatile tools in machine learning and a major building block for various real-world applications with generative AI.
Using BigQuery’s powerful integration with Vertex AI, Google Cloud’s unified AI development platform, we can conveniently create embeddings and calculate similarities right within our data warehouse.
To get our project set up for embedding creation, we did two one-time steps:
- First, we followed these instructions to create a Cloud resource connection in BigQuery called “vertex-connection” and grant the associated service account access to Cloud Storage (as a Storage Object Viewer) and Vertex AI (as a Vertex AI User).
- Next, we created a remote model in BigQuery over Vertex AI models, specifically using Cloud’s latest text embedding model as of this writing, text-embedding-004. Following the example here:
CREATE OR REPLACE MODEL `ice_cream_products.text-embedding-004`
REMOTE WITH CONNECTION `us.vertex_connection`
OPTIONS (endpoint = 'text-embedding-004');From this point forward, we won’t show all the code details behind each step, but rather point you to this GitHub repository with the BigQuery SQL code used to create and output the product comparisons and recommendations we’ll go through below.
Embedding the Right Things
So if our goal is to use embeddings to find similar products across our catalog, the next question is which specific data fields do we choose to create embeddings for?
Initially, it may seem logical to start with the product images and use multimodal embeddings to create vectors for comparing pictures. But in this case, just looking through the images shows an issue with that approach: the products for the same brand all look much more alike based on name and packaging than actual content. If the most similar products for a given Ben & Jerry’s flavor are just other Ben & Jerry’s packages that look the same — but contain very different types of ice cream — then that won’t capture the spirit of similarity we are looking for.
Instead, we focus on embedding two of our previously generated outputs from the initial GenAI asset creation: short product descriptions and review summaries. In both cases, the prompts we used to create these helped strip out many of the brand intricacies or other unique aspects that weren’t core to the actual ice cream products and how customers reviewed them, generating descriptions and summaries that are more properly comparable for products across different brands.
Similar to the vital steps of data exploration/preparation in data science and feature engineering in machine learning, this “pre-processing” step of embedding the “right things” for our problem before building our similarity search is crucial for success in creating valid recommendations.
Below is the SQL code to create product description embeddings, using BigQuery’s ML.GENERATE_EMBEDDING function to turn the description into a 768-dimensional vector for each product (768 is the default output dimensionality, which can be modified).
SELECT
key,
brand_and_product_name,
content AS short_description,
ml_generate_embedding_status AS short_description_embedding_status,
JSON_VALUE(ml_generate_embedding_statistics, "$.token_count") AS
short_description_embedding_token_count,
JSON_VALUE(ml_generate_embedding_statistics, "$.truncated") AS
short_description_embedding_truncated,
ml_generate_embedding_result AS short_description_embedding,
ARRAY_LENGTH(ml_generate_embedding_result) AS short_description_embedding_length
FROM
ML.GENERATE_EMBEDDING(
MODEL `ice_cream_products.text-embedding-004`,
(
SELECT
key,
brand_and_product_name,
/* Handle when description (from previous Gemini call) is not actually real */
IF(TRIM(generate_short_product_description_result) =
'No valid product description to summarize.',
NULL,
generate_short_product_description_result
) AS content
FROM
`ice_cream_products.products_results`
),
STRUCT(
TRUE AS flatten_json_output,
'SEMANTIC_SIMILARITY' AS task_type
)
)The code to create embeddings for review summaries is similar, and the entire SQL script that generates both embeddings, merges them together by product, and outputs them to a new table called `ice_cream_products.product_embeddings` can be seen here.
Calculating Product Similarity
Once we have our embeddings for each product, we can calculate the “distance” between pairs of products, and use those to calculate similarity. The technique we’ll use for this is cosine similarity, which measures the angle between embedding vectors in multidimensional space. More practically in our use case, it helps identify product descriptions and review summaries that lie in a similar “direction” based on semantic similarity, as shown in the simplified example in the image below.
To compute the similarities at scale, we’ll use vector search in BigQuery. The code snippet below shows how we implement the calculation using the VECTOR_SEARCH function with brute force to find the exact nearest neighbors for each product description. In this case, using a “top_k” value of 241 within the vector search function allows us to compute distances for every possible product pair.
CREATE OR REPLACE TEMPORARY TABLE ProductsWithShortDescriptionEmbeddings AS
(
SELECT
*
FROM
`ice_cream_products.product_embeddings`
WHERE
short_description_embedding_length != 0
);
SELECT *
FROM
/* Filter to products w/ actual short description embedding to avoid errors */
VECTOR_SEARCH(
TABLE ProductsWithShortDescriptionEmbeddings,
'short_description_embedding',
TABLE ProductsWithShortDescriptionEmbeddings,
'short_description_embedding',
/* 241 products in entire dataset - getting distances for all pairs */
top_k => 241,
distance_type => 'COSINE',
options => '{"use_brute_force":true}'
);You could also compute the full set of vector distances by first cross joining to get each product pair and then using the ML.DISTANCE function in BigQuery for each pair of embedding vectors. This is feasible because our example product dataset is relatively small, but in more realistic use cases with many thousands or even millions of items, the VECTOR_SEARCH function with an index scales much better by using approximate nearest neighbors to find the few closest matches for any given item.
The process to compute similarity calculations for the review summaries across all product pairs is very much the same, giving us two similarity metrics — one more direct product-based, the other more “vibes”-based — to use in building out our final recommendations.
Building Our Final Recommendations
The entire SQL script that generates both distances for all product pairs, calculates similarity scores on a 0–100 scale by looking at the percentile of distances across pairs, and ranks each comparable product’s similarity to any given starting product can be seen here. With similarity scores and ranks for all product pairs calculated, we are ready to build our recommendations!
The interactive “Ice Cream Recommendations” dashboard shows the results of all our work. You can pick a single product at top and see the most (or least) comparable products in the table below — filterable by brand, average star rating, and more. For instance, choosing Ben & Jerry’s Chocolate Chip Cookie Dough to start gives three solid chocolate chip cookie dough-based products — including the top two from other brands — as most recommended:
Play around with some more products and you can see some cool patterns like various unique flavors (coffee, cherry, cinnamon, even pumpkin!) finding matches across brands and different types of mix-ins/toppings (peanut butter, brownies, mint, etc.) clustering together. You can even find products where there really aren’t any close matches (based on similarity score): turns out Häagen-Dazs Rum Raisin and Talenti Strawberry Hibiscus Sorbetto may be truly one-of-one products (at least in this data set).
There is one detail we’ve glossed over here: how should we combine the two different similarity scores — one for product descriptions, one for review summaries — into one overall metric to get the top recommendations? There’s no objective weighting for this without further research or analysis of how users actually interact with these recommendations, so we’ve set the default to weigh each score equally (50% each) in the overall similarity score.
That said, users have the ability to modify this option in a dropdown slider toward the bottom of the dashboard that uses Looker Studio’s parameter functionality. So if you want to see how the recommendations change if you weigh description similarity or review summary similarity more, go for it!
Next Steps
The most immediate next step is obviously to find your favorite ice cream and its nearest neighbors from the dashboard and go get some in real life!
Once you’ve satisfied those cravings, check out the SQL code for this post or this poster image analysis Jupyter notebook that shows how to use Gemini models, create embeddings, and do vector search over tables in BigQuery. Find your own inspiration and explore the new possibilities that arise when you combine the capabilities of BigQuery, Vertex AI, embeddings, and vector search today!
