Different Types of Search Explained: AI, Keyword, Hybrid Search & more
You’ve probably searched sometimes and found exactly what you were looking for. Whether it’s for a purchase (transactional), get knowledge (informational), or to find a specific page or location (navigational). We’ve all done all of these searches without knowing what ‘type’ of query they were.
There are different kinds of search methods today which include but are not limited to:
1. Keyword search
2. AI search
3. Hybrid search
4. Multi modal search
5. Multi lingual search
6. Voice search
7. AR seach
We’re going to explain what the different types of search are and give some examples of practical applications.
What is ‘keyword search’?
You type in a few words or phrases related to what you're looking for and the search engine scours the internet for pages containing those keywords. Keyword search is basically matching keywords that can be found on the database to your query and serving those results accordingly.
Once the records are found, the final task is for the engine to rank the results, ensuring that the best matches show up at the top of the list. Again, there are different techniques, for example, statistical ranking based on the frequency of the words matched.
Keyword search is the most basic type of search and typically uses existing models such as ‘BM25’, which is ranking algorithm understand that helps to retrieve search results and rank them.
What is AI search?
There's another, more advanced option called ‘AI search’ or ‘semantic search’. It’s powered by artificial intelligence. With Artificial Intelligence (AI) search the search engine can understand your intent and the meaning behind your search, not just the specific words you enter. It can grasp concepts and contextualize your query to deliver far more relevant results compared to keyword search.
How does AI Search work? Understanding Natural Language Processing
Semantic search uses Artificial Intelligence (AI) and Natural Language Processing (NLP) to understand and respond to search queries in a more natural and contextually aware manner. AI fuels the system's learning and adaptation ability, while NLP enables understanding and language generation. Instead of just matching keywords, AI search tries to grasp the meaning and context behind your query to better derive intent.
For example, if you search for "Apple stock price", an AI search engine will understand you're probably interested in the stock price of the technology company Apple, not the fruit. It can make these inferences based on analyzing huge amounts of data to determine semantic relationships between words and phrases.
AI search also considers synonyms, related terms, and concepts. So if you search for "popular RPG games", it may also return results for "MMORPG games" or "adventure games". It can even handle complex queries with multiple clauses like "What is the most popular MMORPG game of all time?" Unlike keyword search, AI search aims to provide the most useful and relevant results and not just the most popular ones that match your keywords.
Using knowledge graphs, also known as a semantic network, AI search is able to understand the relationships between different entities. A knowledge graph, also known as a semantic network, represents a network of real-world entities—i.e. objects, events, situations, or concepts—and illustrates the relationship between them or the ‘ontological framework’.
In the context of a game, the different values that might be in a gaming knowledge graph are things like the game developers, when it was published, platform the game’s available on, genre of the game etc. A knowledge graph could also contain information such as what are the characters or classes of the game (if it is an RPG for instance). And all this information would be stored in a vector database.
Knowledge graphs were introduced to help answer precise questions. Before knowledge graphs, looking for the character in the game was a three step process: find a page that describes the game and the characters or classes, scan the page for characters, and find your character in the list. Knowledge graphs can display the answer directly.
With AI powering search, the search engine is able to process your query in a way that is similar to natural language and you'll get faster, better results tailored to what really matters to you.
These days, many modern search engines that are using a hybrid of semantic and keyword search.
What is Hybrid Search?
Hybrid search is a technique that combines multiple search algorithms to improve the accuracy and relevance of search results. It uses the best features of both keyword-based search algorithms with vector search techniques, combining both dense and sparse vectors in its model.
Sparse vectors are used in keyword searches generated from algorithms that traditional search engines use like ‘BM25’. Sparse simply means that the vectors have fewer non-zero values.
Dense vectors on the other hand have more non-zero values are are utilized by AI search algorithms.
By leveraging the strengths of the different algorithms, hybrid provides a more effective search experience for users.
Keyword Search vs Semantic Search vs Hybrid Search
There are several features of each type of search to take note of.
From research, we know that hybrid search can produce better outcomes compared to a pure ‘keyword’ or even a purely semantic search model. The drawbacks of keyword search being that while it is able to accurately find matching queries based on keywords, it doesn’t always fully understand the context and intent of the query, thus impacting the relevance of the search.
Since AI or semantic search can learn and improve its understanding over time, it is naturally a more powerful model to search on to produce more highly relevant search results. AI search on the other hand, might have trouble understanding simple keywords if the keywords don’t make sense as part of a sentence or a phrase.
AI search personalizes your search results and provides recommendations based on what the model has learned you like. This is extremely powerful for search as it means that users will get more relevant results and search engines become tools that not only provide answers but can engage with users in a more contextualized way.
Hybrid Search Models: Algolia, Cohere and more
Since the advent of AI search, several search engines have emerged as the forerunners of the hybrid/AI search engine. Some of these search engines are not available to consumers but to businesses that require powerful search capability for their databases. They show the world what a AI-powered search future could look like.
One of the leading B2B ‘search as a service’ platforms is Algolia. With its large customer base (over 17,000 customers) across the world spanning e-commerce megastores, to MNC enterprises, Algolia powers many websites and businesses’ search. They have ‘plug-and-play’-like ability for the keyword search using APIs, and AI search capability using the hybrid search model ‘Neuralsearch’ product. Algolia has plenty of resources and documentation to enable businesses to get started on their platform easily.
Cohere creates powerful embedding models that developers can build search applications on top of. They have large language models that companies can use to train on their databases. They provide solutions for enterprises in a flexible, scalable way. Aside from their semantic search product, they have also developed ‘Coral’, which is a knowledge assistant for enterprises.
AI search engines & traditional keyword search engines that use AI
ChatGPT, which stands for Chat Generative Pre-trained Transformer, is large language model -based chatbot developed by Open AI. Launched in November 2022, it has amazed users by its ability to understand complex questions and requests. It’s capable of generating human-like text based on context and past conversations. Open AI’s chatbot and search engine was trained on billions of parameters over time, enabling it to understand more contexts and different type of queries more easily.
Bing, Microsoft's search engine, also utilizes AI and machine learning. Bing's algorithms get better at understanding search intent and providing relevant results the more people use the engine. Some of Bing's AI-powered features include:
· Intelligent answers: Bing can provide direct answers to questions without needing to click through to other websites.
· Visual search: Bing's image search uses AI to detect objects, scenes, and actions in photos to provide more relevant results. You can also search the web using a photo you upload.
· Recommendations: Bing provides recommended searches and results based on what's trending and your own search history. The recommendations get smarter over time as Bing's AI learns your interests.
The New Bing search uses a more conversational approach where you can chat and get answers to complex queries.
Google Search
Google's search engine uses AI and machine learning to better understand the intent behind your queries. When you search for something on Google, its algorithms try to determine what type of information would be most helpful for you based on the context of your search. Google's AI can understand complex questions and commands, allowing you to search in a more natural, conversational way. They also have Bard, which is still in experimental phase, but allows you to search or discovery new topics in a conversational manner.
Introducing Splore
Splore is a search engine for gamers.
We looked at the gamer journey and realized that a lot of the time want answers that traditional search engines can’t answer straight away.
There’s a plethora of content on the web, but a lot of this content is in walled gardens like Discord. A lot of it is also in Reddit forums, which take a long time to read, or simply aren’t indexed by traditional search engines. It’s just hard to find in general unless you really know where to go.
Splore started with a simple keyword search model in Discord and is exploring alternative models to find the best way to understand contextually what gamers are searching for.
Multi-modal Search
What is Multi-modal search?
Multi-modal search refers to searching for different modes of content apart from text based or semantic search. There’s also image search (such as Google lens), and video search (uploading a video into a search engine) and more. Text to image search with Google’s Vertex AI Search model is also one of the most powerful applications of deep learning models is to build an embedding space. This is essentially a map of meanings for texts, images, audio, etc.
How does multi-modal search work?
For example, with an image model, images with similar appearance and meaning will be placed closely together in the embedding space. The model can map an image to an embedding, which is a location in the space. Therefore, if you look around the embedding, you can find other images with similar appearance and meaning. This is how image similarity search works.
With Google’s text to image function, searches are performed without using tags, titles or other descriptions for the search. The results are retrieved purely by looking at the images with the search model.
Potential use cases for this type of search expand beyond pure e-commerce to include realms such as gaming, where users can take a screenshot of an item in a game and use that to search for items that they may want, or even to find out more. Text to image search functionality could also work when you can describe in words what you’re looking for but can’t name the asset, bringing a whole new world of possibility to search.
Multilingual search
Multi-lingual search encompasses search capability that is not limited to one language in the query and results. To build such systems, traditionally, you would have to build individual pipelines for each language. This means additional costs as the system is inefficient. There are two types of multilingual models searches: multilingual and cross-lingual.
Cross-lingual searches enable users to find results in a language that is different to the one they’re using to search. This means a larger variety of relevant search results in different languages can be produced.
Companies like Cohere have created embedding models that can handle over a hundred languages in the same model.
Multi-lingual search presents new opportunities to search for information in new ways.
Voice search
Voice search use automatic speech recognition (ASR) technology. Think of applications like Apple’s Siri, Amazon’s Alexa and Google assistant. All of these use some form of voice recognition that transforms voice into text. Over time, as natural language processing technology improves and large language model (LLMs) are more and more commonly used, applications can be built on top of these technologies. Check out Vocode and Daily.co – startups that are innovating in this space to build interesting applications.
AR search
Lastly, if you’ve heard of Augmented Reality (AR), you might know that of its future applications within search. Google recently updates its maps feature to include AR search functionality, providing information to users who can use their phones to make use of this AR feature.
Conclusion
So there you have it, the key differences between traditional keyword search, AI-powered search and even hybrid search. AI search is the way of the future and will only continue to get smarter over time. While keyword search has served us well for decades, AI search understands language and intent in a way that opens up a whole new world of possibilities. Hybrid search on the other hand will continue to open up new ways to provide more accurate and relevant search results to end users.
