Generative AI Based Search Engine on Azure Cloud
A high — level Solution architecture for Generative AI based Search engine on Azure Cloud
A search engine is a powerful tool that empowers business user with knowledge about the product, connects business product with customer search. For any digital business to be successful there are two primary components.
- A great search engine
- A great recommendation engine
👆 are the reason, why amazon is number 1 in e-commerce space.
In this post, I have outlined a high-level solution for an e-commerce search engine, leveraging Azure, Generative AI and open-source vector database.
Primarily there are two types of search engine in morden times, which helps retrieving desired information from a database based on user search request (User request can be in any form; such as text, voice and image) This article focuses on text based user search input.
- Index based search
- Semantic search
Index-based search relies on creating keyword-based indexes of documents and matching search queries to those indexes, while semantic search uses advanced NLP techniques to understand the user’s intent and context, leading to more accurate and relevant search results.
In this article I have provided the architecture of semantic search and explain each working components and use.
A Generative AI-based Search Engine is a sophisticated system that leverages generative artificial intelligence techniques to generate embeddings for each product. It extracts these embeddings from product descriptions and stores them in a vector database. The system then generates embeddings for user search queries in real-time, matching the user’s search with product descriptions. It captures semantics from the embeddings and provides the best semantic match for the user.
More about Vector DB: With the rise of LLM vector DB is gaining momentum again. Vector is efficient in store and querying embeddings.
What is Embedding: Embedding is a process to convert text / image into numeric value for computer program to understand.
Why Embedding: You cannot pass text as it is into machine learning algorithm to analyse. TF-IDF is a simple form of embedding used for text data.
Embedding by LLM model: LLM embedding are more accurate semantically, because it assign number to text based on contextual similarity.
References: AWS , Microsoft Azure, Google and Open AI docs
Show some support — follow, clap, or do whatever’s cool when you see this. Thanks! 🙌
Follow me on https://www.linkedin.com/in/balarampanda/
