Improve your Tabular Data Ingestion for RAG with Reranking

Boost your RAG system’s accuracy by adding a reranker to select the most relevant context chunks.

Photo by shawnanggg on Unsplash

By Eduardo Rojas Oviedo with Ezequiel Lanza

In our previous post, Tabular Data, RAG, & LLMs, we explored how to improve a large language model’s (LLM’s) ability to generate responses by feeding it small tabular data in various formats to provide necessary context. However, when the provided context is only partially correct, mismatches can lead to less accurate responses from our LLM.

In a typical RAG architecture, the retriever part gets chunks of data based on a similarity search from documents stored in a knowledge base. But are those chunks always the most relevant for our case?

Using the example from our previous article, what if we ask, “Who are the top billionaires in the tech industry in 2024?” and we get documents related to the overall list of world billionaires, including those from various industries that are not specific to tech? If the retriever returns documents about just any world billionaire working in any industry, we won’t get the response we need when the LLM is prompted.

To provide addtional context beyond basic text-based input based on ingesting small tabular data in various formats. It has shown to be useful in enhancing LLMs comprehension not only in plain text but also in other formats like tables.

In this post, we’ll demonstrate how to reduce mismatches by using a ranker that can select the most relevant context chunks, ensuring that the LLM gets the best possible information for generating accurate answers.

Extracting and Processing Data for a RAG System

Our demonstration builds a Q&A chatbot on top of what is considered a basic RAG solution, but we’re adding a reranker at the end to perform an additional ranking over the retrieved chunks. As shown in Image 1, this process has three stages: Data Preparation, Indexing, and Retrieval. We won’t cover the “staging” part; we can consider it as our knowledge base, and it’s not part of this tutorial. For this example, the “knowledge base” will be a PDF containing the World ‘s Billionaires data from Wikipedia, when in a business scenario it could be the entire company database.

Image 1: A comprehensive data pipeline diagram illustrating the stages of data processing, from raw data ingestion to data retrieval, created by the author.

Data Preparation

First, we need to prepare our data for storage in our database. Since our data has both tables and text, the data pipeline creation will follow two paths: one for the text, and one for the tabular data.

In the text Path, we’ll extract the text from the PDF and perform pre-processing tasks, which includes data cleaning, as discussed in a previous article, and implementing a character-based chunking strategy as well as adding in the metadata.

For the tabular path, we’ll extract two sets of tabular data and demonstrate how to convert the information into useful context chunks for later model consumption (as explained in this article). Because the PDF includes tables, we’ll also need to select which approach to use to convert the information to vectors. Then we’ll employ two approaches: one based on row-by-row chunks of information and another using the entire table, as we explained in our previous article Tabular Data, RAG, & LLMs.

Indexing

The next step is to choose how we’ll store the data (which we converted to vectors) Depending on how we want to store the data, the most common approach is to use a unified context collection (UCC). UCC keeps all the information and metadata in a single data location, along with the vectors we’ll use for semantic search; we can think of this approach as having one dataset containing all the information. This is called a collection.

Image 2: Adaptation by the authors from Chorma — Home page. It illustrates Chroma’s Unified Context Collection, showcases the integration of documents, metadata, and embeddings for comprehensive data representation.

Another approach is to use a distributed context collection (DCC) where information, metadata, and vectors are stored in separate collections according to the type of information. In this scenario, we’ll have multiple collections rather than one, central location.

Image 3. Adaptation by the authors from Chroma — Home page comparing Chroma’s Distributed Context Collection and Distributed Table Collection, highlighting their respective components: Document, Metadata, and Embedding.

Each approach has its pros and cons. A DCC may be more complex to manage due to the need for data synchronization across multiple nodes, managing potential inconsistencies,and distributed storage efficiently but it can offer greater scalability and fault tolerance, enabling the system to handle higher loads and recover more quickly from node failures. By contrast, a UCC offers simpler data management by centralizing all data in a single location, ensuring consistent and straightforward administration, but may face challenges with scalability and can become a single point of failure. Below, we evaluate both scenarios to determine how this configuration could affect the retrieval process.

Retrieval

The final stage of this process is retrieving the data. In this stage, the most relevant documents are retrieved from our collections before prompting the LLM, as shown in image four below. A similarity search is performed to retrieve the documents most similar to the user’s question. This sounds great, but are these documents the most relevant? We propose adding an additional step that will perform a verification of the retrieved documents to rank them by similarity. A rerank approach involves scoring the retrieved documents to prioritize the most relevant ones before they are fed into the LLM for response generation. By ensuring the most pertinent information is prioritized, reranking minimizes irrelevant or less useful data, thereby improving the overall performance and reliability of the RAG system. You can explore a deeper explanation of this topic in this Pinecone article.

Image 4. Adaptation by the author from Chorma — Home page and API and API and re-ranking.This diagram illustrates the reranking process. Data points (queries, answers) are adjusted in a semantic space based on relevance.

Let’s see it in action

Let’s then run our experiments! We’ll guide you through setting up the environment, defining functions, and running the experiment. Our goal is to see how well the model performs at answering questions based on a PDF document.

Prepare the environment

We first set up the environment using Python 3.12.3 and provided all the required dependencies in requirements.txt[Install packages in a virtual environment] file with the following required library specifications:

tqdm==4.66.4
spacy==3.7.4
pypdf==4.2.0
langchain-community==0.2.1
langchain-text-splitters== 0.2.0
flashrank==0.2.5
opencv-python==4.9.0.80
camelot-py==0.11.0
ghostscript==0.7
openai 1.30.5
chromadb==0.5.0
sentence-transformers==3.0.0

Convert the Data to PDF

Next, prepare your we need to prepare our data, and load it from a local PDF file. For this purpose, we’ll convert the World’s Billionaires data from Wikipedia, which you can find here: The World’s Billionaires to a PDF format.

from pathlib import Path

# PDF file path
ROOT = Path(os.getcwd()).absolute()
file_path = os.path.join(ROOT, "temp", "World_Billionaires_Wikipedia.pdf")
file_path

Helper Functions for Data Preparation

We now need to define the helper functions we’ll be using later. We’ll start by defining the data preparation functions that convert our data from the initial PDF format into useful chunks. In a previous article, we discussed the essential skills for data cleaning, emphasizing the importance of effective data transformation.

• unicode_to_ascii: Converts Unicode text to ASCII by normalizing Unicode characters, removing any accents or special characters that don’t exist in ASCII, and ensuring the text remains readable in UTF-8 encoding.

import unicodedata
def unicode_to_ascii(text):
"""Normalize unicode values"""
 return unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')

• Load_PDF: This function extracts text from our PDF file and then normalizes it. It also generates metadata values (see T-RAG: Lessons from the LLM Trenches for other approaches), and finally, employs a basic character-based chunking strategy. We’ll use some libraries from LangChain to retrieve information from our PDF (PyPDFLoader), perform the chunking process(RecursiveCharacter TextSplitter), and handle data in a standardized manner as a Document.

from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.documents import Document
def load_pdf(file_path, chunk_size, chunk_overlap):
    
    # load pdf with Langchain loader
    loader = PyPDFLoader(file_path)
    documents = loader.load()
    
    # get total pages count
    page_count = len(documents)
    
    # text cleaning and normalization
    for document in tqdm(documents):
        # text cleaning
        document.page_content = normalize(document.page_content)

        # add metadata 'text' classification
        document.metadata["page"] = str(document.metadata["page"] + 1)
        document.metadata["type"] = "text"
        
    # create text chunks base on character splitter
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    text_chunks = text_splitter.split_documents(documents)
    
    return text_chunks, page_count

• get_tables: This function extracts tables from a PDF document (path) across multiple pages, cleans up table data, generates summaries for each row, and returns structured outputs: tables_summary for textual summaries, metadata_result for metadata, and tables_result for cleaned table data (classified as “row” and “table” as we saw in our previous article Tabular Data, RAG, & LLMs. Our main challenge consists of extracting tabular information using camelot-py and then expressing the information as context chunks that add value to our contextual information. We will need camelot-py to extract tabular data from our PDF.

— Note: Keep in mind the limitation of this algorithm, which does not consider tables distributed across pages.

import Camelot

def get_tables(path: str, pages: int, year: int):
    
    tables_result, metadata_result, tables_summary = [], [], []

    for page in tqdm(range(1, pages)):
        
        table_list = camelot.read_pdf(path, pages=str(page))

        for tab in range(table_list.n):
            
            df_table = table_list[tab].df.dropna(how="all").loc[:, ~table_list[tab].df.columns.isin(['',' '])]
            df_table = df_table.apply(lambda x: x.str.replace("\n", " ").replace("\xa0", " "))

            df_table = df_table.rename(columns=df_table.iloc[0]).drop(df_table.index[0]).reset_index(drop=True)

            if df_table.shape[0] <= 3 or df_table.eq("").all(axis=None):
                continue

            df_table["Year"] = year # Complete missing values (for demonstration purposes only)
            metadata_table = {"source": path, "page": str(page), "year": str(year), "type": "row"}

            df_table["summary"] = df_table.apply(
                lambda x: " ".join([f"{col}: {val}, " for col, val in x.items()]).replace("\xa0", " "),
                axis=1
            )

            docs_summary = [Document(page_content=row["summary"].strip(), metadata=metadata_table) for _, row in df_table.iterrows()]

            tables_result.append(df_table)
            metadata_result.append(metadata_table)
            tables_summary.extend(docs_summary)

            metadata_table = {"source": path, "page": str(page), "year": str(year), "type": "table"}
            tables_summary.append(Document(page_content=df_table.to_markdown(), metadata=metadata_table))
            metadata_result.append(metadata_table)

            year -= 1 # auxiliary code (for demonstration purposes only)

    return tables_summary, metadata_result, tables_result

• normalize : This function takes takes a sentence and makes it easy to analyze by converting special characters to simpler forms, converting all text to lowercase, and then removing unnecessary words and symbols like punctuation, short words, and web addresses. It returns a cleaned-up version of the sentence that’s ready for further processing. Since the example will use text, we need to use a library that can understand language (in this case, English). Then, we’ll use spaCy to load an English NLP model (en_core_web_sm).

import spacy
from spacy.cli import download

try:
    nlp = spacy.load('en_core_web_sm')
except OSError:
    print("Model not found. Downloading the model...")
    download('en_core_web_sm')
    nlp = spacy.load('en_core_web_sm')

def normalize(sentence):
    """Normalize the list of sentences and return the same list of normalized sentences"""
        
    # Normalize Unicode characters to ASCII
    sentence = unicode_to_ascii(sentence)
    
    # Convert the sentence to lowercase and process it with spaCy
    sentence = nlp(sentence.replace('\n', ' ').lower())
    
    # Lemmatize the words and filter out punctuation, short words, stopwords, mentions, and URLs
    sentence_normalized = " ".join([word.lemma_ for word in sentence if (not word.is_punct)
                                    and (len(word.text) > 2) and (not word.is_stop) 
                                    and (not word.text.startswith('@')) and (not word.text.startswith('http'))])
    return sentence_normalized

Helper Functions for the Indexer

Next, we’ll create functions for the indexing stage. We’ll use chromadb as our embedding database because it’s open source . As mentioned before, we will be using both UCC and DCC scenarios.

Unified Context Scenario

In this scenario, we’re leveraging ChromaDB to create a unified collection of structured data, optimizing it for efficient semantic search, and embedding generation.

The key feature highlighted is metadata={“hnsw:space”: “cosine”} with which we configure the distance function used in semantic search and the embedding provider employed. By default, ChromaDB uses the Sentence Transformers all-MiniLM-L6-v2 model to create embeddings.

import chromadb
clientdb = chromadb.PersistentClient()

def unified_context_collection(text_chunks, tables_summary):
    # generate structure for unified vector scenario
    
    # generated chromadb collection
    unified_collection = clientdb.create_collection(name="unified_context_collection", metadata={"hnsw:space": "cosine"})

    # Unified 'page_content' and 'metadata'
    unified_docs = [doc.page_content for doc in text_chunks + tables_summary]
    unified_meta = [doc.metadata for doc in text_chunks + tables_summary]

    # generate unique identifiers
    unified_ids = [str(uuid.uuid4()) for _ in unified_docs]

    # index data and generate default embeddings (vectors)
    unified_collection.add(documents=unified_docs, metadatas=unified_meta, ids=unified_ids)

    return unified_collection

Distributed Context Scenario

In this distributed scenario configuration, we utilize ChromaDB to manage two distinct collections: distrctx_context_collection and distrtbl_table_collection. Each collection is optimized for semantic search and embedding generation using the cosine similarity distance function ({“hnsw:space”: “cosine”}). It processes text_chunks for contextual data and tables_summary for structured tabular data, assigning unique identifiers and indexing them for efficient data management.

import chromadb
clientdb = chromadb.Client()

def distributed_context_collection(text_chunks, tables_summary):
    # generate structure for unified vector scenario
    
    # generated chromadb collection
    distrctx_collection = clientdb.create_collection(name="distrctx_context_collection", metadata={"hnsw:space": "cosine"})
    distrtbl_collection = clientdb.create_collection(name="distrtbl_table_collection", metadata={"hnsw:space": "cosine"})

    # Distributed context 'page_content' and 'metadata'
    distrctx_docs = [doc.page_content for doc in text_chunks]
    distrctx_meta = [doc.metadata for doc in text_chunks]

    # generate context unique identifiers
    distrctx_ids = [str(uuid.uuid4()) for _ in distrctx_docs]


    # Distributed Table 'page_content' and 'metadata'
    distrtbl_docs = [doc.page_content for doc in tables_summary]
    distrtbl_meta = [doc.metadata for doc in tables_summary]

    # generate table unique identifiers
    distrtbl_ids = [str(uuid.uuid4()) for _ in distrtbl_docs]

    # index data and generate default embeddings (vectors)
    distrctx_collection.add(documents=distrctx_docs, metadatas=distrctx_meta, ids=distrctx_ids)
    distrtbl_collection.add(documents=distrtbl_docs,metadatas=distrtbl_meta, ids=distrtbl_ids)

    return distrctx_collection, distrtbl_collection

The Helper Function for Reranking

Now, we’ll look at the retrieval functions. For the reranker, we’ve chosen FlashRank for its ease of use and becauseit operates independently of Torch or Transformer frameworks, running efficiently on CPUs with a minimal memory footprint of approximately 4MB. FlashRank plays a pivotal role in our RAG system by reranking candidate chunks after a semantic search. Using SOTA cross-encoders and advanced models, FlashRank evaluates the relevance of retrieved chunks based on their similarity to the input query. The function will generate the metric “score” (max_score, min_score, avg_score, std_score) that offers insights into the quality and distribution of the ranked results, helping to assess and refine the performance of the ranking process.

Note: The final segment of metrics was added for demonstrative purposes only, and final evaluation is not required in the proposed production setting.

from flashrank import Ranker, RerankRequest
import pandas as pd
pd.set_option('display.max_colwidth', None)


def get_ranked_chunks(query: str, collectionContext, test_scenario=3, year_scope=2024, top_results=20, top_rank: int=5):
    
    conditions = {
        0: {"year": str(year_scope)},
        1: {"$and": [{"type": "row"}, {"year": str(year_scope)}]},
        2: {"$and": [{"type": "table"}, {"year": str(year_scope)}]},
        3: {"$and": [{"year": str(year_scope)}, {"$or": [{"type": "table"}, {"type": "row"}]}]},
    }

    results = collectionContext.query(
        query_texts=[query],
        n_results=top_results,
        where=conditions[test_scenario]
    )

    total_collection_results = len(results)
    
    passages = [
        {"id": results["ids"][0][i], "text": results["documents"][0][i], "meta": results["metadatas"][0][i]}
        for i in range(total_collection_results)
    ]

    ranker = Ranker()
    rerankrequest = RerankRequest(query=query, passages=passages)
    ranked_results = ranker.rerank(rerankrequest)
    total_rank_results = len(ranked_results)

    chunks = [item['text'] for item in ranked_results[:top_rank]]
    total_chunks = len(chunks)

    df_ranked = pd.json_normalize(ranked_results).rename(columns=lambda x: x.replace('meta.', ''))
    df_ranked = df_ranked.sort_values(by='score', ascending=False)

    metrics = {
        "max_score": "{:+.6f}".format(df_ranked['score'].max()),
        "min_score": "{:+.6f}".format(df_ranked['score'].min()),
        "avg_score": "{:+.6f}".format(df_ranked['score'].mean()),
        "std_score": "{:+.6f}".format(df_ranked['score'].std()),
        "types": ', '.join(df_ranked['type'].unique()),
        "search_count": str(total_collection_results),
        "rerank_count": str(total_rank_results),
        "chunks_count": str(total_chunks),
        "chunks": ', '.join(df_ranked['text'].str[:7])
    }

    new_row_dict = {"Query": query, "Collection": collectionContext.name, "Scenario": test_scenario, **metrics}
    
    return chunks, new_row_dict, df_ranked

LLM API Client Configuration

We’re almost done! We still need to configure the last step to inference using the LLM with the context we prepared in previous sections.

Depending on your specific LLM setup, you should adjust the configuration accordingly for your personal use.

from openai import AzureOpenAI

client = AzureOpenAI(
    api_key = os.environ.get("OAI_API_Key"), 
    api_version = os.environ.get("OAI_API_Version"), 
    azure_endpoint = os.environ.get("OAI_API_Base"))

MESSAGE_SYSTEM_CONTENT = """You are a customer service agent that helps a customer with answering questions. 
Please answer the question based on the provided context below. 
Make sure not to make any changes to the context, if possible, when prepare answers so as to provide accurate responses. 
If the answer  not be found in context, just politely say that you do not know, do not try to make up an answer."""

def response_test(question:str, context, model:str = "gpt-4"):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "system",
                "content": MESSAGE_SYSTEM_CONTENT,
            },
            {"role": "user", "content": question},
            {"role": "assistant", "content": '\n '.join(context)},
        ],
    )
    
    return response.choices[0].message.content

Perform the Test

Now that we have all the functions in place, let’s perform the test following the steps below:

• 1. Get the chunks: We selected a chunk size of 600. With this, we handle 34 pages and process 129 chunks (96 for chunk_size 800, 77 for 1000). Experimenting with these parameters is quite entertaining!

text_chunks, page_count = load_pdf(file_path, chunk_size=600, chunk_overlap=100)
print(f"Total text chunks: {len(text_chunks)}")
print(f"Total pages: {page_count}")

• 2. Get the tabular data:

tables_summary, metadata_result, tables_result = get_tables(file_path, page_count, 2024)
print(len(tables_result))

Note: The value “2024” is a temporary solution that facilitates adding functional value to our classification metadata.

• 3. Put the data into collections (UCC / UCD)

# Get the unified context data collection
unified_collection = unified_context_collection(text_chunks, tables_summary)

# Get the distributed context data collections
distrctx_collection, distrtbl_collection = distributed_context_collection(text_chunks, tables_summary)

Finally, here is the code snippet that allows us to control our tests. As you can see, our proposal encompasses: extraction and reranking of context information (get_ranked_chunks), prompt LLM (response_test), and finally, in ‘df_results’, we accumulate the metrics.

import pandas as pd
pd.set_option('display.max_colwidth', None)

def query_controler(questions, unified_collection, distrtbl_collection):
    
    df_results = pd.DataFrame(columns=[
        "Query", "Collection", "Scenario", "Answer",  
        "max_score", "min_score", "avg_score", "std_score", 
        "types", "search_count", "rerank_count", "chunks_count", "chunks"
    ])
    
    def process_query(query_source, collection, year_scp):
        chunks, new_row_dict, df_ranked = get_ranked_chunks(query=query_source, collectionContext=collection, year_scope=year_scp)
        response = response_test(query_source, chunks)
        new_row_dict["Answer"] = response
        
        return new_row_dict
    
    for key in questions.keys():
        query_source = questions[key]["query"]
        year_scp = questions[key]["year_scope"]
        
        new_row_dict = process_query(query_source, unified_collection, year_scp)
        df_results.loc[len(df_results)] = new_row_dict
        
        new_row_dict = process_query(query_source, distrtbl_collection, year_scp)
        df_results.loc[len(df_results)] = new_row_dict
        
    return df_results

We’ve now reached the most anticipated moment! Can we learn the answer to: “Who was the fourth millionaire of 2022?” using both scenarios (UCC and DCC)

questions = {}
questions["question1"] = {"query": "Who was the fourth millionaire of 2022?", "year_scope": 2022}

df_results = query_controler(questions, unified_collection, distrtbl_collection)
df_results[["Collection", "Answer", "max_score", "min_score", "types", "rerank_count", "chunks_count","chunks"]]

Image 5: Results table generated by the RAG application after processing a user query.

As you can see on Image 5, using both collections, “unified_context_collection” and “distrtbl_table_collection,” they both produced similar outputs answering our initial question. Since we configured metrics, this helps identify the quality of chunks retrieved. In both scenarios, we received scores ranging from +0.000086 to +0.000017, which indicates that the retrieved chunks consistently align closely with the query’s requirements. This range of scores suggests that the information retrieved from both collections is consistently relevant and closely matches what we sought to find, providing reliable and credible data.

For this case, we selected collections that include rows and tables, with seven rerank counts and five final contextual chunks each. Overall, the analysis confirms Bill Gates’ ranking and net worth with high precision and consistency across both data collections.

We’d like to show you one additional example. In this scenario, we’ve removed the metrics to facilitate understanding the results. However, it’s necessary to emphasize question number 3: “Based on context, which Billionaries have Google as a source of their fortune in 2021?” which offers an interesting opportunity to see that this question was resolved using only row-based table chunks.

questions = {}
questions["question2"] = {"query": "What's the Elon Musk net worth by 2023 year?", "year_scope": 2023}
questions["question3"] = {"query": "Based on context, which millionaires in 2021 have Google as the source of their fortune?", "year_scope": 2021}

df_results2 = query_controler(questions, unified_collection, distrtbl_collection)
df_results2[["Query", "Answer", "types", ]]

Image 6: Results table generated by the RAG application after processing a user query.

What Did We Learn?

In this tutorial for ingesting small tabular data while working with LLMs, we explored various aspects crucial for understanding and implementing RAG frameworks. From data extraction and normalization to semantic search and reranking, each step plays a pivotal role in achieving accurate and contextually rich responses in natural language processing tasks.

We also highlighted the challenges and opportunities of integrating tabular data into RAG frameworks. Notably, we emphasized the importance of chunking strategies and the impact of irrelevant context on model performance, drawing attention to relevant resources for deeper exploration.

This study not only enhances our understanding of RAG frameworks but also provides practical insights for implementing them effectively in real-world scenarios. As the field of natural language processing continues to evolve, such investigations pave the way for more sophisticated and contextually aware language models.

If you would like to learn more, here are some suggestions for further reading:

• Density-based retrieval relevance. This is vital in retrieval-augmented generation, where irrelevant context can confuse generative models and degrade performance. Unlike relational databases, vector search systems return the nearest neighbors regardless of relevance.
• Large Language Models Can Be Easily Distracted by Irrelevant Context. The authors show that irrelevant information significantly reduces model performance. They also identify mitigation strategies, including self-consistent decoding and prompt instructions for ignoring irrelevant data.
• Open Platform for Enterprise AI (OPEA): When deploying a RAG system, we will face multiple challenges such as ensuring scalability, handling data security, and integrating with existing infrastructure. This open-source project helps with the deployment by providing robust tools and frameworks designed to streamline these processes and facilitate seamless integration.

About the Authors

Eduardo Rojas Oviedo, Platform Engineer, Intel

Eduardo Rojas Oviedo is a dedicated RAG developer within Intel’s dynamic and innovative team. Specialized in cutting-edge developer tools for AI, Machine Learning, and NLP, he is passionate about leveraging technology to create impactful solutions. Eduardo’s expertise lies in building robust and innovative applications that push the boundaries of what’s possible in the realm of artificial intelligence. His commitment to sharing knowledge and advancing technology drives his ongoing pursuit of excellence in the field.

Ezequiel Lanza, Open Source AI Evangelist, Intel

Ezequiel Lanza is an open source AI evangelist on Intel’s Open Ecosystem team, passionate about helping people discover the exciting world of AI. He’s also a frequent AI conference presenter and creator of use cases, tutorials, and guides to help developers adopt open source AI tools. He holds an MS in data science. Find him on X at @eze_lanza and LinkedIn at /eze_lanza

Follow us!

Medium, Podcast, Open.intel , X , Linkedin