Revolutionizing Information Retrieval: The Role of Large Language Models in a Post-Search Engine Era
Large language models promise to transform information retrieval, generating nuanced answers rather than links. This article explores the implications of LLMs ushering in a new post-search paradigm.
Introduction
For more than 20 years, search engines have served as the go-to method for discovering information online.
With the exponential growth of the Internet and the vast expansion of accessible data, search engines have been facing challenges in delivering their ability to retrieve the most relevant and useful information for users.
Due to this, since the beginning of this year there has been an escalating curiosity surrounding the prospects of Large Language Models (LLMs) such as OpenAI GPT-4, Anthropic’s Claude 2, or Inflection’s PI, as potential game-changers, capable of revolutionsing information retrieval, analysis and consumption.
While traditional search engines will not disappear entirely, they are likely to transition into “crawling“ and “ranking” engines that work in tandem with LLMs to find and rank content, data, and information in regards to the context the user is exploring.
LLMs are pioneering a more conversational way of interacting with information, fundamentally changing the user experience around finding and consuming and editing digital content.
Background
Before starting, let’s spend a few introductory words about Large Language Models for those who are not very familiar with them or need a brief refresh.
Large Language Models (acronym: LLMs) represent a category of Artificial Intelligence models that employ deep learning algorithms to comprehend and produce language that closely resembles human language.
Large language models possess an astonishing range of capabilities that are transforming how we interact with computers and electronic devices. Their application include text generation, question answering, summarisation of extensive texts, data labeling and text classification, language translation, code aiding, and many other applications.
Progress in fields such as Machine Learning (ML) and Natural Language Processing (NLP), advancements in Computer Hardware, such as the introduction of tensor processing units (TPUs) and also the optimisation of neural network architectures have allowed these models to attain a deep comprehension of language, making them exceptionally good at processing and generating human-like text.
This confluence of technological advancements doesn’t merely make them faster or more efficient, but inherently more nuanced and accurate in both understanding our language and formulating their responses, leading them to be ‘intelligent’ in their very own way.
They can synthesise insights across diverse datasets that were used to train them, integrating disparate information sources into unified understanding.
One of their most profound strengths is the ability to rapidly acquire new expertise simply by training on additional data, expanding their knowledge and competencies without losing prior learning.
With their versatile, comprehensive language abilities and capacity for instant learning, LLMs are pioneering new possibilities for information technology.
Over the course of this year, diverse emerging models with varying degrees of efficiency and output quality have surfaced from the efforts of both private companies and the open source community; nonetheless, in general, the results yielded by these models are astounding and possess the potential to revolutionise various industries.
Information Retrieval Beyond Search Engines
Traditional search engines, as we know them today, are primarily designed to process explicit queries or keywords and retrieve relevant information based on those queries.
They rely on crawlers (or spiders or bots), indexes and ranking mechanisms to analyse web pages, rank their content, and determine their relevance to specific search queries.
Search engines aim to provide users with a list of ranked results that best match their queries, often using factors such as keyword relevance, page authority, and user engagement metrics to determine the ranking.
However, with the advent of Large Language Models and their ability to comprehend natural language queries and generate personalised responses, there is a growing recognition of the limitations of traditional search engines.
LLMs can synthesise long textual data that makes sense to the reader and remains relevant to their original query.
These models learn their skills by first studying massive amounts of information. They get good at understanding the context; much like how we pick up the overall meaning of a sentence from the words that compose it.
(Note: I can write more in depth articles about the above topic if it’s of any interest)
They also use previous learnings to help with new tasks, learnings that are usually based on information created by humans [1].
LLMs provide more accurate and tailored results which can be further explored by engaging in an actual conversation with the model.
Given this development, search engines may soon find themselves relinquishing their long held position as gatekeepers and “front pages” of the Web.
This change could lead to the emergence of user interfaces based on the combination of a search engine and a Large Language Model, as we are already seeing with Bing Chat.
The essential role of a search engine in this novel architecture is not only to crawl and index the Internet, but also to find and present the most relevant information to the LLM, based on the specific user needs and preferences interpreted by the model.
In this way, search engines would no longer directly show users the ranked results of a query. Instead, they would send the most pertinent information to the Language Model (perhaps supported by URLs), which they can then synthesise, summarise, and present in a more consumable way for the end user.
The search engine’s crawling capabilities allow it to scan and catalog massive amounts of web content. Its ability to then filter and rank this content for relevance provides the optimal information for the LLM to absorb and transform into useful responses.
Together, the crawling and ranking functions of the search engine work seamlessly with the synthesis capabilities of the LLM to create a vastly improved information discovery process compared to traditional search.
By applying semantic and contextual understanding to its corpus of indexed web content, the model is equipped to synthesise the most relevant ranked information into a response matching the user’s query, response which could include web links as a reference.
This transition acknowledges the power and potential of LLMs in understanding natural language and generating personalised responses, while also recognising that ranking engines are essential for organising and filtering information effectively.
By combining the strengths of LLMs and search engines, the goal is to deliver highly relevant and personalised results that align with the user’s intent, context, and preferences in a way that could be digested easier than ever.
This transition from search engines to some sort of ranking engines that act in the back-end of Language Models would show a shift from a singular focus on information retrieval to a broader emphasis on information synthesis and visualisation.
LLMs would provide the capabilities for understanding and generating language, while ranking engines would select and retrieve information for the LLMs so that it can be presented to the user in an easy way.
Advantages of LLMs over Search Engines as the front page of the Web
A fundamental benefit of LLMs over search engines lies in their capacity to comprehend intricate queries and produce personalised responses. Unlike search engines, which depend on explicit queries or keywords for information retrieval, LLMs possess the capability to grasp and interpret natural language queries, resulting in more precise and pertinent replies.
LLMs can also tailor their responses based on user preferences and prior interactions, enhancing the usefulness and relevance of the information provided.
Another advantage offered by LLMs is related to their capacity to learn and adapt to novel tasks and domains through their existing knowledge.
Search engines are confined by fixed algorithms and data structures, which can restrict their flexibility and adaptability.
Conversely, LLMs possess the ability to assimilate new data and user interactions, enabling them to enhance their accuracy and relevance progressively as time unfolds.
Thanks to their capability to extract relevant chunks of information from documents, LLMs utility can potentially expand beyond web-based content.
By efficiently parsing and analysing vast volumes of textual data, LLMs can extract key insights, summarise complex documents, and provide concise and pertinent information to users.
This ability to leverage and distill information from various document sources (e.g. PDFs, .csv, .md, .txt and so on), extends their utility beyond simple knowledge extraction becoming actual productivity engines.
To further elevate the discussion on versatility, it is worth highlighting the immense potential of integrating LLMs with diverse software systems.
This can potentially empower users to interact with their software suite in a conversational way in order to effortlessly execute specific tasks.
This integration paves the way for a truly conversational and streamlined user experience, offering the advantage of automating numerous tasks, including creating and polishing textual data, arranging appointments, or performing in-depth data analysis.
The automation capabilities introduced in this way can significantly save time and resources for both individuals and organisations, optimising their workflows and enhancing overall productivity.
Ultimately, LLMs possess the ability to cater to a diverse range of languages, opening numerous possibilities in terms of accessibility of content and information for a global audience.
Users can engage with LLMs in their preferred language, even if the underlying information the model has been trained on was not initially available in that specific language. This inclusive attribute aids in dismantling language barriers and can greatly improve information accessibility for individuals who speak different languages. No matter the language pages are written in, the Language Model can analyse them and automatically present results in the language selected by the user.
Challenges and Limitations of Large Language Models
While LLMs hold significant promise, they are not without their fair share of challenges and limitations. Foremost among these is the accuracy and validity of the information they present.
LLMs generate responses based on their training data, which may occasionally lack accuracy, currency and may be affected by bias. Consequently, there may be a risk of users being exposed to erroneous or incomplete information. However, basing the response on information crawled from the web, should in theory mitigate this risk (unless, indeed, such information is based on fake news, false statements, misleading content and so on).
Privacy concerns represent another significant challenge tied to LLMs.
The data employed for training LLMs could potentially include personal or sensitive information, thereby raising apprehensions regarding data privacy and security.
In addition to this, the data generated during interactions between LLMs and users may also give rise to privacy concerns over sensitive data. Safeguarding user privacy and ensuring secure handling of data emerge as vital considerations within the context of LLM utilisation.
Lastly, current LLMs, despite their impressive capabilities, still have limitations when it comes to the amount of data they can process in a single instance.
Think of it like reading a book: while humans might skim or skip parts to grasp the overall message, an LLM is designed to process specific chunks of information at once.
This is because of memory and computational constraints inherent in their design.
Just as a person can only hold so many thoughts in their mind at once, an LLM has a ‘working memory’ dictated by its architecture, specifically called ‘context window’ or in simple terms, the maximum amount of words or characters it can consider simultaneously.
This limitation means that, for very long or complex queries, the LLM might not fully capture or comprehend all aspects presented to it.
As a result, the responses generated can sometimes be oversimplified, lacking in detail, or even potentially misrepresentative of the original query’s intent.
It is worth noting that there are emerging prompt engineering techniques being developed to mitigate these constraints.
However, at the current state, these techniques can be quite technical and may require a deeper understanding of the model’s inner workings to be effectively utilised, therefore, they are not readily for the general public.
As research progresses, there is hope that architectures will evolve to further enhance the capabilities and reliability of LLMs.
Closing Thoughts
LLMs offer the remarkable potential to extract content and information and reconstruct it in a manner that can be tailored to the unique needs of each user. Irrespective of factors such as time availability, attention span, language barriers, and other constraints, LLMs can curate and present information in a highly personalised and human-centric way.
This ability, when coupled with the search engines functions of crawling and ranking information from the Internet, has the potential to alleviate the pain points and enhance the overall user experience, when it comes to navigating web and consuming digital content.
It is still essential to acknowledge and address the challenges and limitations that accompany LLMs. Ensuring the accuracy and validity of information presented, safeguarding privacy, and improving their architectures are critical areas that require continuous attention and improvement.
As LLMs continue to evolve and overcome these challenges, their capacity to adapt and optimise information delivery to individual users’ preferences and constraints is poised to reshape the way we interact with and consume information both at work and in our free time.
Footnotes
[1] In theory synthetic data, or data generated by an AI, can be used to train a Large Language Model. However, this may lead to unintended results and a degradation of their performance. Read more here.
Attributions
Icons from Freepik, Becris, Dinosoft Labs, Design Circle and Smash Icons.
Article images were generated with Midjourney and are part of a synthetic art collection that I’m developing.
