From Keywords to Understanding: The Evolution of Search — Part 1
Why Do We Need Search?
Let’s start with a simple question: Can you imagine a world without search? It would be like trying to find a needle in a haystack, but without the haystack! Search has become an integral part of our daily lives. Whether it’s finding a recipe,booking a flight, or researching a topic, search engines are our go-to helpers.
But what exactly is happening behind the scenes when you type something into a search bar? Let’s break it down,
The Basics of Search: Finding What You Need
At its core, search is about finding relevant information within a vast amount of data. Think of it as a digital librarian,categorizing and organizing information so you can easily find what you’re looking for.
From Keywords to Understanding: The Evolution of Search
Traditionally, search relied heavily on keywords. If you wanted to find information about “cats,” you’d type “cats” into the search bar. This is called lexical search. While effective for simple queries, it often fell short when it came to understanding the nuances of human language.
Enter semantic search. This approach goes beyond keywords by understanding the meaning and context of your query. It’s like having a conversation with a knowledgeable friend who understands what you’re really asking. For instance, searching for “best friend movies” might not only show movies about friendship but also romantic comedies or dramas with strong friendship themes.
Let’s explore these search methods in more detail:
Content-Based (Lexical) Search: Matching Words
Content-based search primarily focuses on matching keywords or exact phrases provided by the user. It operates on the principle of identifying documents or data that contain the exact terms queried.
Here are different methods within content-based search:
Keyword Search
Keyword search is one of the most basic and widely used methods for retrieving information from databases, websites, document repositories, and various other types of digital content. This method relies on matching specific words or phrases provided by the user to locate relevant documents or data entries.
Example
When you type “digital cameras” into a search engine, you’re essentially asking it to find and retrieve pages or products that contain the exact phrase “digital cameras”. And most likely, you might see listings for various digital camera models, reviews, or articles related to digital photography.
Limitations
While keyword search is simple, fast and scalable — one of the severe limitations includes their efficacy when supplied with typos (for instance, searching for “dgtel camra” will fail to fetch the products of pages matching “digital camera”).
Fuzzy Search
Fuzzy search is a specialized search technique designed to improve search accuracy by accommodating variations in spelling, typographical errors, and approximate matches to the query terms. Unlike traditional exact match keyword searches, fuzzy search algorithms allow for flexibility in matching similar but not necessarily identical terms.
Example
When you type “smrt phne” into a search engine, the fuzzy search algorithm calculates the edit distance between “smrt phne” and potential matches in the indexed data. It identifies that “smart phone” is a close match, despite the typo. The search engine retrieves results that include “smart phone,” ensuring that relevant documents or entries are not missed due to minor spelling errors.
Limitations
Fuzzy search eliminates the inefficiency of the Keyword Search by handling typographical errors and variations in spelling, enhancing search accuracy. One of the limitations include understanding context or intent behind the query, as it primarily focuses on string similarity.
Full-Text Search
Full-text search is a powerful and versatile search technique that scans entire documents or datasets to locate relevant information based on the query terms. Unlike keyword-based searches that focus on specific fields or exact matches, full-text search examines the complete content of the documents to provide comprehensive search results.
Example
Imagine you’re using a search engine to find a document containing information about “climate change impacts on coastal cities”, the search engine indexes the entire content of documents, including every occurrence of words like “climate,” “change,” “impacts,” “coastal,” and “cities.” The search engine looks through the indexed data to find documents where these terms appear and retrieves documents that contain these terms, potentially showing excerpts where the query terms appear frequently or in prominent positions, such as in titles or abstracts.
Limitations
Full-text search is a great step forward (when compared with keyword search) in finding and retrieving the information even if the search term itself isn’t used. Implementing and maintaining full-text search systems can be more complex compared to simpler keyword-based searches.
How about the others?
We shall review the other search methods in the next part.
