E-Commerce with Generative AI using OpenAI and Azure
Azure OpenAI and Semantic Kernel
In my initial experience with ChatGPT and its paid version, ChatGPT Plus, I was astounded by their ability to produce code based on my prompts. These prompts varied from planned product requirements to informal brainstorming about web design aesthetics. I managed to develop about 80% of an ASP.NET Web microservice architecture complete with comprehensive Authentication & Identity management and a frontend Blazor application with a modern look and feel. The remaining 20% involved a combination of some challenges with the tool and my own expertise to fulfill the tasks. You can refer to my earlier blog post where I detail this journey.
Once I gained access to the OpenAI API, the prospect of using this capability through custom application code was exciting. While my initial interactions were somewhat challenging, my attention was soon drawn to Microsoft’s open-source Semantic Kernel (SK).
Semantic Kernel describes itself as a “lightweight SDK facilitating the integration of AI Large Language Models (LLMs) with traditional programming languages.” In essence, it simplifies coding with LLMs such as GPT 3.5 & 4 using this library. As of the time of writing, Semantic Kernel SDK is only available for Python & C#.
Virtual Grocer Assistant Demo
With the collaboration of my colleagues, we dedicated about a week to constructing a demonstration making use of this library, coupled with an LLM and some Azure services, to display the potential of such an architecture in an e-commerce scenario. Our demonstration was coined the “Virtual Grocer Assistant”.
Our goal was to empower users to locate their desired shopping items via a text-based input powered by a chat like interface. For instance, the user could type: “I wish to purchase olive oil, eggs, and milk.” and the response would present the available items in the store, offering the option to add these items to their shopping cart. Additionally, we desired to incorporate a fun feature: recipe suggestions. Given the immense training data of LLMs, fulfilling such requests became significantly more feasible.
Let’s explore some example prompts:
Architecture
Now let’s unpack the architecture we used.
We opted for Blazor .NET 7 WASM as our frontend technology, hosted on an ASP.NET Web API. Then, we employed Azure’s OpenAI service, a managed platform for OpenAI’s LLMs, accessible via an API Key. The consumption charge is based on the number of tokens used. You can find more details about the pricing here: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/
Tokens, as defined by Microsoft, are “the fundamental units of text or code that an LLM AI employs to process and generate language.” Both the user’s input and the LLM’s output text are divided into these tokens for processing. GPT-4, which is the latest version at the time of writing, has made significant strides in processing human languages. In this particular demo, we used the “text-davinci-003” LLM.
We also incorporated Azure Cognitive Search to manage our product inventory. In a real-world setting, we might also use another persistent store like a relational database (Postgres / SQL Server, etc) for additional inventory information. With a structured response from our LLM, we could construct a search query for our Cognitive Search database to retrieve the available products in stock.
Lastly, we utilized Azure Blob Storage containers to house our product image data, which is then referenced by our frontend.
Prompt Engineering
Working with Large Language Models (LLMs) requires a shift in mindset — mastering Prompt Engineering is crucial. Ensuring that you’ve clearly set the expectations for the LLM before providing input to it is pivotal; otherwise, coding based on its response could prove challenging.
In our demonstration, we direct our LLM via our semantic skill to anticipate either recipes, ingredients, or a combination of both, and to deliver a well-structured JSON response. We then convert a portion of this response into a Lucene query, which we run against our Azure Cognitive Service index containing our inventory data. This process enables us to merge the vast knowledge of the LLM with our private data to provide a relevant response to the end user.
Token constraints can be a tough nut to crack. The Davinci model we used allows roughly 4000 tokens per response. If a response exceeds this limit, it’s truncated. Although there are methods to retrieve the subsequent set of tokens (response text), it’s an aspect to bear in mind and, in most situations, probably limit. This ensures you don’t end up delivering a 100-page response to the end user and avoid overspending on consumption. It’s also important to note that models may count tokens differently.
Now let’s look at the ASP.NET Web API side of the code. Initially, it calls Azure OpenAI through the Semantic Kernel library. Apart from the end user’s prompt, we also pass our semantic and native skills to facilitate the processing of the request. The semantic skill is forwarded to the Semantic Kernel library to formulate the prompt sent to the text-davinci-003 model. Following the model’s response, which includes some JSON, our native skill is invoked. At this juncture, we transform the JSON portion of the response into a fuzzy search Lucene query for use in calling Cognitive Search. Lastly, we combine the result from Cognitive Search and the unstructured response from the LLM to create our own structured response sent back to the client.
Here’s the prompt we utilized in our “PersonalShopper” semantic skill. With more practice and the introduction of newer, more advanced models, we anticipate honing this further.
For each item in the input, classify it as a shopping-list-item or a recipe. Examples:
I want a chocolate cake | shopping-list-item
I want oil | shopping-list-item
I'd like a chocolate cake | shopping-list-item
I want to make chocolate cake | recipe
I want to bake a chocolate cake | recipe
I'd like to make my own chocolate cake | recipe
I need some milk | shopping-list-item
I need chocolate cake | shopping-list-item
I want to buy salsa | shopping-list-item
I want pre made chocolate cake | shopping-list-item
I want cookies from the bakery | shopping-list-item
If nothing at all in the input can be classified as a shopping-list-item, recipe, or something you would buy at a grocery store, say something humorous in response to the input, then say you're sorry this is a grocery store. But, if at least one thing in the input can be classified, then it is OK.
For a recipe, find the ingredients for the a popular recipe for that item and show the result as a JSON object with the following properties:
- RecipeName
- RecipeUrl
- RecipeDescription
- Ingredients
- Name
- Quantity
- Directions
Note, the Directions should be an array.
For a shopping-list-item, show the result as a JSON object with the following properties:
- Name
All results should be returned in a JSON array.
++++
{{$input}}
++++Let’s look at an example of the raw response returned from the model when typing in “I want to make chocolate cake and buy salsa”.
Sorry, this is a grocery store, so we can't help you with flying to the moon. Here are the results for the items you mentioned:
[
{
"RecipeName": "Chocolate Cake",
"RecipeUrl": "https://www.allrecipes.com/recipe/10813/best-chocolate-cake/",
"RecipeDescription": "This is a very easy and delicious chocolate cake you can make in no time. It's moist and rich, and the flavor is out of this world!",
"Ingredients": [
{
"Name": "all-purpose flour",
"Quantity": "2 cups"
},
{
"Name": "granulated sugar",
"Quantity": "2 cups"
},
{
"Name": "baking soda",
"Quantity": "1 teaspoon"
},
{
"Name": "salt",
"Quantity": "1 teaspoon"
},
{
"Name": "unsweetened cocoa powder",
"Quantity": "3/4 cup"
},
{
"Name": "vegetable oil",
"Quantity": "1 cup"
},
{
"Name": "buttermilk",
"Quantity": "1 cup"
},
{
"Name": "eggs",
"Quantity": "2"
},
{
"Name": "vanilla extract",
"Quantity": "2 teaspoons"
},
{
"Name": "boiling water",
"Quantity": "1 cup"
}
],
"Directions": [
"Preheat oven to 350 degrees F (175 degrees C). Grease and flour two 9 inch round pans.",
"In a large bowl, mix together the flour, sugar, baking soda, salt and cocoa.",
"In a separate bowl, combine the oil, buttermilk, eggs and vanilla. Mix into the dry ingredients until just blended.",
"Stir in the boiling water last. Batter will be thin. Pour evenly into the prepared pans.",
"Bake 30 minutes in the preheated oven, or until a toothpick inserted into the center of the cake comes out clean."
]
},
{
"Name": "salsa"
}
]In this case, we got a semi-structured response with some text at the top and a JSON object under it. Future versions of this will allow the entire response to always be fully structured.
Next our native C# “QueryBuilderSkill” is called to form a fuzzy search Lucene query:
[SKFunction("BuildQuery")]
[SKFunctionName("BuildQuery")]
public string BuildQuery(string item)
{
var jsonStartIndex = item.IndexOf("[");
if (jsonStartIndex == -1)
{
jsonStartIndex = item.IndexOf("{");
}
var itemJson = item.Substring(jsonStartIndex, item.Length - jsonStartIndex);
var unstructuredModelResponse = jsonStartIndex == 0 ? string.Empty : item.Substring(0, jsonStartIndex).Replace("Here are your results:", string.Empty).Trim();
var json = JsonNode.Parse(itemJson);
var subQueries = json switch
{
JsonObject obj => BuildQueryForObject(obj),
JsonArray arr => BuildQueryForArray(arr),
_ => Enumerable.Empty<string>()
};
var recipes = json switch
{
JsonObject obj => new List<Recipe> { BuildRecipeForObject(obj) },
JsonArray arr => BuildRecipesForArray(arr),
_ => new List<Recipe>()
};
recipes.RemoveAll(r => string.IsNullOrEmpty(r.Name));
var serializedRecipes = JsonConvert.SerializeObject(recipes);
var query = string.Join(" or ", subQueries.Distinct(StringComparer.InvariantCultureIgnoreCase));
return $"{unstructuredModelResponse}{StringExtensions.ModelResponseDelimiter}{serializedRecipes}{StringExtensions.ModelResponseDelimiter}{query}";
}Here’s the Lucene syntax outputted from our native skill.
"all-purpose flour"~ or "granulated sugar"~ or "baking soda"~ or "salt"~ or "unsweetened cocoa powder"~ or "vegetable oil"~ or "buttermilk"~ or "eggs"~ or "vanilla extract"~ or "boiling water"~ or or "salsa"~Our Cognitive Search index is populated with the following product catalog data:
{
"value": [
{
"@search.action": "upload",
"id": "1",
"name": "Organic Unbleached White All-Purpose Flour",
"aliases": ["white flour", "all-purpose flour", "organic flour", "unbleached flour", "baking flour"],
"size": "5 lb",
"image_name": "flour.jfif",
"cost": 6.79
},
{
"@search.action": "upload",
"id": "2",
"name": "Premium Pure Cane Granulated Sugar",
"aliases": ["granulated sugar", "white sugar", "sweetener", "baking sugar"],
"size": "4 oz",
"image_name": "sugar.jfif",
"cost": 4.79
},
{
"@search.action": "upload",
"id": "3",
"name": "Cocoa Powder Unsweetened",
"aliases": ["cocoa powder", "chocolate powder", "baking cocoa", "unsweetened cocoa"],
"size": "10 oz",
"image_name": "cocoa-powder.jfif",
"cost": 7.29
},
{
"@search.action": "upload",
"id": "4",
"name": "Baking Powder",
"aliases": ["rising agent", "leavening agent", "baking ingredient"],
"size": "8.1 oz",
"image_name": "baking-powder.jpg",
"cost": 2.49
},
{
"@search.action": "upload",
"id": "5",
"name": "Acme Pure Baking Soda",
"aliases": ["sodium bicarbonate", "baking soda", "cleaning agent", "leavening agent"],
"size": "1 lb",
"image_name": "baking-soda.jpg",
"cost": 1.29
},
{
"@search.action": "upload",
"id": "6",
"name": "Acme Natural Sea Salt",
"aliases": ["sea salt", "cooking salt", "table salt", "seasoning"],
"size": "26 oz",
"image_name": "salt.jpg",
"cost": 3.53
},
{
"@search.action": "upload",
"id": "7",
"name": "DeLallo Instant Espresso Powder",
"aliases": ["espresso powder", "coffee powder", "instant coffee", "coffee flavor"],
"size": "1.94 oz",
"image_name": "espresso-powder.webp",
"cost": 6.99
},
{
"@search.action": "upload",
"id": "8",
"name": "Acme 2% Reduced Fat Milk",
"aliases": ["2% milk", "reduced fat milk", "low fat milk", "dairy"],
"size": "1 gal",
"image_name": "milk.jfif",
"cost": 2.79
},
{
"@search.action": "upload",
"id": "9",
"name": "Acme Extra Virgin Olive Oil",
"aliases": ["olive oil", "cooking oil", "extra virgin olive oil", "EVOO"],
"size": "32 oz",
"image_name": "olive-oil.jpg",
"cost": 13.99
},
{
"@search.action": "upload",
"id": "10",
"name": "Acme Large Brown Eggs",
"aliases": ["eggs", "large eggs", "brown eggs", "protein"],
"size": "12 ct",
"image_name": "eggs.jfif",
"cost": 1.89
},
{
"@search.action": "upload",
"id": "11",
"name": "Vanilla Extract",
"aliases": ["vanilla", "baking extract", "flavor extract", "vanilla flavor"],
"size": "4 oz",
"image_name": "vanilla-extract.jfif",
"cost": 14.77
},
{
"@search.action": "upload",
"id": "12",
"name": "Gilbert's All Natural Caprese Chicken Sausage",
"aliases": ["sausage", "chicken", "breakfast sausage", "caprese"],
"size": "4 ct / 10 oz",
"image_name": "chicken-sausage.webp",
"cost": 3.99
},
{
"@search.action": "upload",
"id": "13",
"name": "Acme Organic Baby Spinach",
"aliases": ["spinach", "greens", "salad greens", "vegetable", "leafy greens"],
"size": "5 oz",
"image_name": "baby-spinach.jpg",
"cost": 3.49
},
{
"@search.action": "upload",
"id": "14",
"name": "Acme Shredded Mild Cheddar Cheese",
"aliases": ["cheddar cheese", "shredded cheese", "mild cheddar", "cheese"],
"size": "8 oz",
"image_name": "cheddar-mild-cheese.jfif",
"cost": 2
},
{
"@search.action": "upload",
"id": "15",
"name": "Acme Salsa Verde Medium",
"aliases": ["salsa", "medium salsa", "verde salsa", "green salsa", "Mexican sauce"],
"size": "16 oz",
"image_name": "medium-salsa.jfif",
"cost": 3.02
},
{
"@search.action": "upload",
"id": "16",
"name": "Acme Salsa Mild",
"aliases": ["salsa", "mild salsa", "chunky salsa", "Mexican sauce"],
"size": "16 oz",
"image_name": "mild-salsa.jfif",
"cost": 3.49
},
{
"@search.action": "upload",
"id": "17",
"name": "Medium Ripe Avocado",
"aliases" : ["ripe avocado", "medium avocado", "avocado"],
"size": "1 ct",
"image_name": "large-avocado.jfif",
"cost": 0.99
},
{
"@search.action": "upload",
"id": "18",
"name": "Acme Pure Ground Black Pepper",
"aliases" : ["mccormick pepper", "ground pepper", "black pepper"],
"size": "3 oz",
"image_name": "black-pepper.jfif",
"cost": 5.49
},
{
"@search.action": "upload",
"id": "19",
"name": "Acme Organic Cilantro",
"aliases" : ["organic cilantro", "simple cilantro", "cilantro"],
"size": "0.5 oz",
"image_name": "cilantro.jfif",
"cost": 2.29
},
{
"@search.action": "upload",
"id": "20",
"name": "Roma Tomato",
"aliases" : ["italian tomato", "red tomato", "tomato", "tomatoes", "roma tomatoes"],
"size": "1 ct",
"image_name": "roma-tomato.jfif",
"cost": 0.45
},
{
"@search.action": "upload",
"id": "21",
"name": "Acme Petite Canned Diced Tomatoes",
"aliases" : ["red tomatoes", "diced tomatoes", "canned tomatoes"],
"size": "14.5 oz",
"image_name": "diced-canned-tomatoes.jfif",
"cost": 1.00
},
{
"@search.action": "upload",
"id": "22",
"name": "Jalapeno Peppers",
"aliases" : ["jalapeno", "hot pepper", "jalapeno pepper", "jalapeno peppers"],
"size": "1 ct",
"image_name": "jalapeno-pepper.jfif",
"cost": 0.22
},
{
"@search.action": "upload",
"id": "23",
"name": "Jumbo Red Onions",
"aliases" : ["red onions", "jumbo onions", "onion", "onions"],
"size": "1 ct",
"image_name": "jumbo-red-onion.jfif",
"cost": 0.70
},
{
"@search.action": "upload",
"id": "24",
"name": "Garlic",
"aliases" : ["fresh garlic", "garlic clove", "garlic cloves"],
"size": "1 ct",
"image_name": "garlic.jfif",
"cost": 0.59
},
{
"@search.action": "upload",
"id": "25",
"name": "Acme Chili Powder",
"aliases" : ["chili powder", "smart chili"],
"size": "1.98 oz",
"image_name": "chili-powder.jfif",
"cost": 1.25
},
{
"@search.action": "upload",
"id": "26",
"name": "McCormick Ground Cumin",
"aliases" : ["mccormick cumin", "ground cumin"],
"size": "1.5 oz",
"image_name": "ground-cumin.webp",
"cost": 3.79
},
{
"@search.action": "upload",
"id": "27",
"name": "Large Limes",
"aliases" : ["green limes", "lime", "lime juice"],
"size": "1 ct",
"image_name": "large-lime.jfif",
"cost": 0.59
},
{
"@search.action": "upload",
"id": "28",
"name": "Honeycrisp Apple",
"aliases" : ["sweet apple", "apple"],
"size": "1 ct",
"image_name": "honeycrisp-apple.jfif",
"cost": 1.50
},
{
"@search.action": "upload",
"id": "29",
"name": "Acme Apple Juice",
"aliases" : ["juice", "apple juice"],
"size": "64 fl oz",
"image_name": "apple-juice.jfif",
"cost": 2.69
},
{
"@search.action": "upload",
"id": "30",
"name": "Fresh Atlantic Salmon Whole Fillet Farm Raised",
"aliases" : ["salmon fillet", "atlantic salmon", "salmon", "salmon fillets"],
"size": "1 ct",
"image_name": "fresh-atlantic-salmon.jpg",
"cost": 24.98
},
{
"@search.action": "upload",
"id": "31",
"name": "Bakery Fresh Chocolate Cake",
"aliases" : ["cake", "chocolate cake", "bakery cake", "bakery chocolate cake", "whole chocolate cake", "premade cake", "premade chocolate cake"],
"size": "6 in / 16.7 oz",
"image_name": "bakery-chocolate-cake.jfif",
"cost": 5.00
},
{
"@search.action": "upload",
"id": "32",
"name": "Bakery Fresh Oatmeal Raisin Cookies",
"aliases" : ["cookies", "oatmeal raisin", "raisin cookies", "bakery cookies", "oatmeal cookies"],
"size": "16 ct",
"image_name": "oatmeal-raisin-cookies.jfif",
"cost": 4.49
},
{
"@search.action": "upload",
"id": "33",
"name": "Bakery Fresh Chocolate Chip Cookies",
"aliases" : ["cookies", "chocolate chip", "chocolate chip cookies", "chocolate cookies", "bakery cookies"],
"size": "16 ct",
"image_name": "chocolate-chip-cookies.jfif",
"cost": 4.49
},
{
"@search.action": "upload",
"id": "34",
"name": "Betty Crocker Super Moist Chocolate Fudge Cake Mix",
"aliases" : ["chocolate cake mix", "chocolate fudge mix", "cake mix", "fudge cake", "moist cake", "moist chocolate cake"],
"size": "15.25 oz",
"image_name": "betty-crocker-super-moist-chocolate-fudge-cake.jfif",
"cost": 1.79
}
]
}Here is the full ASP.NET Web API endpoint logic:
[HttpPost]
public async Task<ChatMessage> Post([FromBody]ChatPrompt prompt)
{
var skContext = await _semanticKernel.RunAsync(prompt.Prompt!, _invSkill["PersonalShopper"], _querySkill["BuildQuery"]);
if (skContext.ErrorOccurred)
{
return new ChatMessage
{
PreContent = skContext.Result,
IsError = true
};
}
//Parse out both the recipe collection and lucene query from the LLM model + native QueryBuilder skill response
var skContextResultParts = skContext.Result.Split(StringExtensions.ModelResponseDelimiter, StringSplitOptions.RemoveEmptyEntries);
var unstructuredResponseExists = skContextResultParts.Length == 3;
Recipe[] recipes;
string? query;
string unstructuredResponse = null;
if (unstructuredResponseExists)
{
unstructuredResponse = skContextResultParts[0];
recipes = JsonConvert.DeserializeObject<Recipe[]>(skContextResultParts[1]);
query = skContextResultParts[2];
}
else
{
recipes = JsonConvert.DeserializeObject<Recipe[]>(skContextResultParts[0]);
query = skContextResultParts[1];
}
//Call Cognitive Search passing in the lucene query
var cog = new CogSearch(_config);
var products = cog.SearchProducts(query);
//Construct response object for our consumer
return new ChatMessage
{
PreContent = unstructuredResponse,
Products = products,
Recipes = recipes,
IsUser = false,
RecipeContent = recipes?.Any() ?? false ?
"Here are some recipe details" :
string.Empty,
InventoryContent = products?.Any() ?? false ?
"These are items we have in stock related to your ask." :
"We don't have any of the required ingredients in stock."
};
}At the end, we return a ChatMessage that our Blazor frontend can parse for rendering.
Alternatives
While, I made use of exclusively Microsoft and .NET technologies, there are other options out there. These tools at the time of this demo were the easiest ones for me to get started with. I could’ve used an Angular or React frontend, but sharing common objects between Blazor and my ASP.NET Web API made me go that route.
We tried some of the other available models in Azure OpenAI such as gpt-3.5 and text-ada-001. I did not get access to gpt-4 yet which I am eager to try. Google has their PaLM 2 model which would be interesting to work with in the future.
As for the Generative AI platform, there are other options and their capabilities are constantly getting improved. GCP’s Generative AI App Builder looks interesting, but I have not had a chance to try it or get access to it yet. I don’t know much about Amazon Bedrock at the moment. There is also HuggingFace and LangChain which look to be promising on the open source side.
Conclusion
In essence, the combined power of OpenAI, Semantic Kernel, and Azure’s services led to the creation of a robust, AI-driven “Virtual Grocer Assistant”. The AI effectively processed user requests, from simple grocery items to complex recipe suggestions, demonstrating the potential of this technology in revolutionizing the e-commerce landscape. As we continue to build upon this foundation, we anticipate a future where AI integration becomes even more seamless and efficient, heralding a new era in digital user experiences.
For those interested in seeing some additional examples using Semantic Kernel in Python or C#, check out their repo here: https://github.com/microsoft/semantic-kernel
