Unleashing the Power of the Information Retriever in the Retrieval Augmented Generation Pipeline

Exploring how an information retriever enhances generation capabilities in the RAG pipeline using different knowledge sources

Information Retrieval. Credit — Nicola Tonellotto Homepage

Artificial intelligence is transforming the way we process and understand text. One popular use case in the Large Language Model (LLM) field is the Retrieval Augmented Generation (RAG). Central to the RAG pipeline is an Information retriever, which fetches relevant documents from a knowledge source to augment the context for generating responses. So, we will focus on the Information retrieval component of the RAG pipeline shown below.

RAG Pipeline

Understanding the Information Retriever

An information retriever serves as a bridge between the input and an extensive knowledge source. It scans a corpus of documents, ranks them according to their relevance to the input, and retrieves the most pertinent documents. The retrieved documents are then passed to a Large Language model, which generates a response based on both the input and the additional context provided by the retrieved documents.

1. Information Retrieval using Apache Solr

The first example of a neural retriever we’ll look at uses Solr as the document store. The solr_retriever function initializes a connection to a Solr instance. It sends a query to the Solr instance and retrieves the top matching documents.

Solr Retriever

The above query is a GET request sent to a Solr instance, which is an open-source search platform. Let’s break down the different components:

• http://000.000.000.00:0000/solr/superknowa/select: This is the URL of the Solr service. 000.000.000.00:0000 is the IP address and port where Solr is running, solris the application context (this is usually ‘solr’), superknowais the name of the collection being queried, and select is the request handler that Solr uses to search the collection.
• q=’+question+: This is the query parameter, where question is the search term that will be used to search the Solr collection. The +question+ is a way to concatenate the variable question into the URL string in Python.
• q.op=AND: This is the default operator for the query. When the operator is AND, a document will only match the query if it contains all of the terms. You can adjust this operator to get the best results depending on the use case.
• wt=json: This parameter determines the response format. In this case, json is specified, so the response will be in JSON format.

So, in simple terms, this Solr query is searching for documents in the superknowa collection that contains all of the terms in the question string and it will return the results in JSON format.

2. Information Retrieval using Elasticsearch

Next, we have an implementation of a neural retriever using Elasticsearch, a popular open-source search and analytics engine. The elastic_retriever function connects to an Elasticsearch instance and retrieves documents based on a given query similar to the Solr query explained in the previous section.

ES Retriever

3. Information Retrieval using Watson Discovery

Finally, we introduce a neural retriever that uses Watson Discovery as the knowledge source. The process_discovery_retriever function initializes a connection to a Watson Discovery instance. It takes a query, sends it to the Watson Discovery instance, and retrieves the top matching entities.

Watson Discovery Retriever

Conclusion

Neural retrievers, whether built with Apache Solr, Elasticsearch, or Watson Discovery, are vital to the Retrieval Augmented Generation pipeline. They enable the model to augment its context with relevant information from a vast knowledge source, leading to more nuanced and insightful responses. By understanding and leveraging the power of Information retrievers, we can push the boundaries of AI communication and create more meaningful interactions with AI models.

The next stage in the RAG pipeline is to implement a Re-ranker to get the most relevant document. Learn more about it in our next blog.

Thanks to Abhilasha Mangal for building the Knowledge Retrieval component of our RAG pipeline. And check our GitHub for full code implementation.

Follow Towards Generative AI for more technical content related to AI.

Subscribe to the 3 min newsletter to learn about 3 most impactful things in Generative AI every week.