CAG or RAG !!!

Cache Augmented Generation

Introduction

The problem with traditional LLMs 🤔:

Large language models like GPT-4 excel at generating text ✍️, but they face critical limitations 🛑:

• Static Knowledge 📚: Their knowledge is frozen 🧊 in time ⏳, unable to access updates post-training8
• Hallucinations 👻: They sometimes invent 🤥 plausible-sounding but incorrect facts.
• Token Limits 📜: Long documents? They struggle to retain context

This is where Retrieval Augmented Generation(RAG) comes in allowing models to leverage knowledge from external source.

Retrieval Process

• 1️⃣ Retrieval 🔎: The system retrieves top-ranking documents or text segments (e.g., from a vector database 🗄️). Think of it like a librarian quickly finding the most relevant books!
• 2️⃣ Augmentation ➕: The system appends the retrieved segments to the user’s query. It’s like adding extra context to your question for a clearer understanding!
• 3️⃣ Generation ✍️: The LLM generates the final output by processing the augmented prompt. 📝 Here the LLM crafts a response using the original query and the retrieved information! ✨

Advantages of RAG:

• 📅 Up-to-date Information ℹ️: RAG stays current by fetching real-time data. No more outdated answers! 🚀
• ⚖️ Smaller LLMs 💡: By storing knowledge externally, the model can be made lighter and more efficient. Less heavy lifting! 💪
• 🔍 Reality Check ✅: RAG reduces hallucinations (false information) by referencing real documents. Bye-bye, made-up facts! 👋

Challenges of RAG:

• ⏳ Latency 🐌: Each query includes an extra retrieval step, which may slow down response time. It’s like waiting in line at the library! 🚶
• ️ Incorrect Data ❌: If irrelevant or outdated documents are retrieved, it may lead to mistakes. Garbage in, garbage out! 🗑️
• ⚙️ System Complexity 🤯: Managing an external index or database can be difficult, especially with frequent updates. It’s like juggling too many balls at once! 🤹

What is CAG ?

Cache-Augmented Generation (CAG) 🚀 is a framework designed to enhance language model performance by preloading relevant knowledge into the model’s context 🧠, leveraging the extended context capabilities of Large Language Models (LLMs). This approach eliminates the need for real-time retrieval during inference ⏱️, offering a streamlined and efficient alternative to Retrieval-Augmented Generation (RAG) 🔄, particularly for tasks where knowledge is stable and predictable ✅.

Instead of dynamically loading or retrieving documents from external sources 📚 as in RAG, CAG processes and preloads the entire knowledge base into the model, allowing it to generate responses from a cached context 💾.

Comparison of Traditional RAG and CAG Workflow

How CAG Works

⚙️The CAG framework operates in three main phases:

1. External Knowledge Preloading 📤: A curated collection of documents relevant to the target application is preprocessed and encoded into a precomputed key-value (KV) cache. This KV cache encapsulates the model’s inference state and is stored for quick access during inference.
2. Inference 🤔: During the inference phase, the precomputed KV cache is loaded alongside the user’s query. The model leverages the cached context to generate responses, thereby eliminating the latency and risks associated with dynamic retrieval processes.
3. Cache Reset ♻️: To sustain system performance, the KV cache can be reset efficiently by truncating newly appended tokens, allowing for rapid reinitialization without the need for a complete reload, ensuring the framework maintains high responsiveness.

Advantages of CAG: 🌟

• 🚀 Low Latency:💨 The system skips the retrieval step for lightning-fast responses! No more waiting around! ⏳➡️⚡
• 🧩 Simplicity: ✨ No need for complicated RAG pipelines with external retrieval and augmentation components. Keep it simple😉
• 🔗 Integrated Context: 🧠 Since the model processes everything from the start, it can improve multi-step reasoning and make more insightful connections! 💡

Potential Issues with CAG: 🤔

• ⚖️ Context Size Limitations: 📏 If your knowledge set is too large, you can’t load it all into the model. It’s like trying to fit too much into a suitcase! 🧳
• ⏱️ Initial Computation: ⏳ The system requires more processing upfront to precompute the KV cache. Think of it as the prep time before a delicious meal! 🍳
• 📰 Stale Data: 📰➡️🗑️ If your data changes frequently (e.g., news), you may need to reload the cache regularly. Out with the old, in with the new! 🔄

Technology Stack

• 🧠Transformers (meta-llama/Llama-3.2–1B-Instruct) model to generate KV Cache
• 🔥Groq (mixtral-8x7b-32768) model to generate response
• 🛠️Langchain — RAG Application Framework
• 🗄️ChromaDB :Vectorstore
• ⚙️Runpod.io — GPU access (NVIDIA-A40,48GB RAM)

Code Implementation

Here I have tried to Implement RAG , and CAG also attempted to combine RAG and CAG.

Install required libraries

%pip install langchain langchain_groq langchain_community 
%pip install langchain_experimental
%pip install pymupdf
%pip install -qU sentence_transformers
%pip install -q -U git+https://github.com/huggingface/transformers.git
%pip install -q -U git+https://github.com/huggingface/accelerate.git
%pip install pymupdf
%pip install chromadb

Download Data

!mkdir ./data
!mkdir ./chunk_caches
!wget "https://www.binasss.sa.cr/int23/8.pdf" -O "./data/fibromyalgia.pdf"

Import Required Dependencies

import pymupdf
from langchain_groq import ChatGroq
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
import json

from langchain_community.embeddings import OpenAIEmbeddings
from langchain_community.chat_models import ChatOpenAI
from langchain_community.embeddings import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel
from langchain_core.runnables import RunnablePassthrough
from langchain_core.runnables import RunnableParallel
import torch
import json
import torch
import argparse
import os
import json
from time import time
from sentence_transformers import SentenceTransformer, util
from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM
from transformers.cache_utils import DynamicCache, OffloadedCache, QuantizedCache
import random
import gc

Load the Data

from langchain_community.document_loaders import PyMuPDFLoader

file_path = "./data/fibromyalgia.pdf"
loader = PyMuPDFLoader(file_path)
#
docs = loader.load()
print(len(docs))
print(docs[0].metadata)
######################Response##############################
{'producer': 'Acrobat Distiller 6.0 for Windows', 'creator': 'Elsevier', 'creationdate': '2023-01-20T09:25:19-06:00', 'source': './data/fibromyalgia.pdf', 'file_path': './data/fibromyalgia.pdf', 'total_pages': 8, 'format': 'PDF 1.7', 'title': 'Fibromyalgia: Diagnosis and Management', 'author': 'Bradford T. Winslow MD', 'subject': 'American Family Physician, 107 (2023) 137-144', 'keywords': '', 'moddate': '2023-02-27T15:02:12+05:30', 'trapped': '', 'modDate': "D:20230227150212+05'30'", 'creationDate': "D:20230120092519-06'00'", 'page': 0}

Generate Semantic Chunking

from langchain_experimental.text_splitter import SemanticChunker
from langchain_huggingface import HuggingFaceEmbeddings
text_splitter = SemanticChunker(HuggingFaceEmbeddings())
#
chunks = text_splitter.split_documents(docs)
#
print(len(chunks)
#
print(chunks[0])
#
chunks[0].metadata
####################################Response##################
35
Document(metadata={'producer': 'Acrobat Distiller 6.0 for Windows', 'creator': 'Elsevier', 'creationdate': '2023-01-20T09:25:19-06:00', 'source': './data/fibromyalgia.pdf', 'file_path': './data/fibromyalgia.pdf', 'total_pages': 8, 'format': 'PDF 1.7', 'title': 'Fibromyalgia: Diagnosis and Management', 'author': 'Bradford T. Winslow MD', 'subject': 'American Family Physician, 107 (2023) 137-144', 'keywords': '', 'moddate': '2023-02-27T15:02:12+05:30', 'trapped': '', 'modDate': "D:20230227150212+05'30'", 'creationDate': "D:20230120092519-06'00'", 'page': 0}, page_content='February 2023 ◆ Volume 107, Number 2\t\nwww.aafp.org/afp\x08\nAmerican Family Physician\u2002 137\nFibromyalgia is characterized by diffuse mus\xad\nculoskeletal pain, fatigue, poor sleep, and other \nsomatic symptoms.1 Chronic diffuse pain affects \n10% to 15% of adults in the general population \nworldwide, many of whom have fibromyalgia.2,3 \nApproximately 2% of people in the United States \nhave fibromyalgia, although the prevalence var\xad\nies across populations and with the diagnostic \ncriteria used.3 Fibromyalgia can occur in chil\xad\ndren and adults and is found worldwide and \nacross cultures. Women are diagnosed more \nfrequently than men;\u200b a Scottish survey found \nthat women are diagnosed between two and 14 \ntimes as often as men depending on the crite\xad\nria used.3,4 Changes in the diagnostic criteria \nover the past decade, including the elimination \nof specific tender points, have resulted in more \npatients with chronic pain meeting the criteria \nfor fibromyalgia.3-5\nPathophysiology\nFibromyalgia is likely caused by disordered cen\xad\ntral nociceptive signal processing that leads to \nsensitization expressed as hyperalgesia and allo\xad\ndynia, which is similar to chronic pain conditions \nsuch as irritable bowel syndrome, interstitial cys\xad\ntitis, chronic pelvic pain, and chronic low back \npain.6,7 Functional brain imaging suggests that \nthis aberrant processing may be attributed to an \nimbalance between excitatory and inhibitory neu\xad\nrotransmitters, particularly within the insula.8 \nSuggested etiologies include dysfunction of the \nhypothalamic-pituitary-adrenal axis and the \nautonomic nervous system, diffuse inflammation, \nglial cell activation, small fiber neuropathy, and \ninfections such as the Epstein-Barr virus, Lyme \ndisease, and viral hepatitis.9 Twin studies suggest \na genetic component may also be a factor.10\nFibromyalgia:\u200b Diagnosis and Management\nBradford T. Winslow, MD, University of Colorado School of Medicine, Aurora, \nColorado;\u200b Swedish Family Medicine Residency, Englewood, Colorado\nCarmen Vandal, MD, and Laurel Dang, MD, Swedish Family Medicine Residency, Englewood, Colorado\n CME  This clinical content conforms to AAFP criteria for \nCME. See CME Quiz on page 127. Author disclosure:\u200b No relevant financial relationships.')
{'producer': 'Acrobat Distiller 6.0 for Windows',
 'creator': 'Elsevier',
 'creationdate': '2023-01-20T09:25:19-06:00',
 'source': './data/fibromyalgia.pdf',
 'file_path': './data/fibromyalgia.pdf',
 'total_pages': 8,
 'format': 'PDF 1.7',
 'title': 'Fibromyalgia: Diagnosis and Management',
 'author': 'Bradford T. Winslow MD',
 'subject': 'American Family Physician, 107 (2023) 137-144',
 'keywords': '',
 'moddate': '2023-02-27T15:02:12+05:30',
 'trapped': '',
 'modDate': "D:20230227150212+05'30'",
 'creationDate': "D:20230120092519-06'00'",
 'page': 0}

Save the chunks as a json file

def generate_chunks(data) -> dict:
    """
        Creating chunks of fixed size and store as a json.
    """
    text = ""
    chunk = 0
    final_dict = {}
    for indx,page in enumerate(data,start=1):
        text = text + "\n" +  page.page_content
        final_dict[f"chunk_{indx}"] = text
        text = ""
    
    with open("./chunks.json", 'w') as file:  
        json.dump(final_dict, file)
        
    return final_dict

chunk_dictionary = generate_chunks(chunks)

Instantiate LLM

from langchain_groq import ChatGroq
import os
os.environ['GROQ_API_KEY'] = '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'
llm = ChatGroq(temperature=0, model_name="mixtral-8x7b-32768")

Summary Prompt

prompt_template = """Write a concise summary of the following:
                    {text}
                    CONSCISE SUMMARY:
                    """

Generate Summaries for each chunk and save json file

def generate_chunk_summaries(chunk_dictionary: dict, prompt: str) -> dict:
    """
        For each chunk, generate summary and store as a json.
    """
    prompt = PromptTemplate(template=prompt_template, input_variables=["text"])
    chain = LLMChain(llm=llm, prompt=prompt)

    summary_dict = {}
    
    for i in range(len(chunk_dictionary)):
        summary = chain.invoke(chunk_dictionary[f"chunk_{i+1}"])
        summary_dict[f"chunk_{i}"] = summary

    with open("./summary.json", 'w') as file:  
        json.dump(summary_dict, file)

    return summary_dict

summary_dictionary = generate_chunk_summaries(chunk_dictionary, prompt_template)

Initialize Tokenizer and Model

#config
HF_TOKEN = <your hf token>
MODEL_NAME_CHILD = "meta-llama/Llama-3.2-1B-Instruct"
#
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME_CHILD, token=HF_TOKEN)
model = AutoModelForCausalLM.from_pretrained(
            MODEL_NAME_CHILD,
            torch_dtype=torch.float16,
            device_map="auto",
            token=HF_TOKEN
        )

Load the chunks dictionary

def load_json_data(data_path: str) -> dict:
    """
        funtion to load a json based on the string path provided
    """
    with open(data_path, 'r') as file:  
        final_dict = json.load(file)
    return final_dict

chunks_dictionary = load_json_data("./chunks.json")

Check for GPU usage

def find_gpu_allocation():
    """
        Function to find the gpu usage
    """
    allocated = torch.cuda.memory_allocated() / 1024**2
    reserved = torch.cuda.memory_reserved() / 1024**2
    print(f"Memory Allocated: {allocated}, Memory Reserved: {reserved}")

Prepare the knowledge for KV Cache

def preprocess_knowledge(
    model: AutoModelForCausalLM,
    tokenizer: AutoTokenizer,
    prompt: str,
) -> DynamicCache:
    """
    Prepare knowledge kv cache for CAG.
    Args:
        model: HuggingFace model with automatic device mapping
        tokenizer: HuggingFace tokenizer
        prompt: The knowledge to preprocess, which is basically a prompt

    Returns:
        DynamicCache: KV Cache
    """
    print("Before Embedding Step:")
    find_gpu_allocation()
    embed_device = model.model.embed_tokens.weight.device
    input_ids = tokenizer.encode(prompt, return_tensors="pt").to(embed_device)
    past_key_values = OffloadedCache()
    with torch.no_grad():
        outputs = model(
            input_ids=input_ids,
            past_key_values=past_key_values,
            use_cache=True,
            output_attentions=False,
            output_hidden_states=False
        )
    print("After Caching Step:")
    find_gpu_allocation()
    result = outputs.past_key_values
    
    # Follow below steps to clean the GPU memory
    outputs.past_key_values = None
    del outputs.past_key_values
    del outputs
    del input_ids
    del embed_device
    del model
    del past_key_values
    torch.cuda.empty_cache()
    gc.collect()
    
    print("After Deletion of Everything Step:")
    find_gpu_allocation()
    
    return result

Store the Generated KV cache data to disk

def write_kv_cache(kv: DynamicCache, chunk):
    """
    Write the KV Cache to a file.
    """
    key_cache = kv.key_cache
    value_cache = kv.value_cache
    original_device = kv.original_device
    torch.save(key_cache, f"./chunk_caches/{chunk}_key.pt")
    torch.save(value_cache, f"./chunk_caches/{chunk}_value.pt")
    torch.save(original_device, f"./chunk_caches/{chunk}_od.pt")
   

Extract the Knowledge from Cache based on the instructions

def prepare_kvcache(documents, answer_instruction: str = None, chunk = None):
    # Prepare the knowledges kvcache

    if answer_instruction is None:
        answer_instruction = "Answer the question with a super short answer."
    knowledges = f"""
    <|begin_of_text|>
    <|start_header_id|>system<|end_header_id|>
    You are an assistant for giving short answers based on given context.<|eot_id|>
    <|start_header_id|>user<|end_header_id|>
    Context information is bellow.
    ------------------------------------------------
    {documents}
    ------------------------------------------------
    {answer_instruction}
    Question:
    Summarize the entire document while keeping all the keypoints intact.
    """
    # Get the knowledge cache
    t1 = time()
    kv = preprocess_knowledge(model, tokenizer, knowledges)
    print("kvlen: ", kv.key_cache[0].shape[-2])
    write_kv_cache(kv, chunk)
    t2 = time()
    return kv, t2 - t1

Create Dynamic Cache

def dynamic_cache_creator(knowledges, chunk):
    answer_instruction = None
    knowledge_cache, prepare_time = prepare_kvcache(knowledges, answer_instruction=answer_instruction, chunk=chunk)
    kv_len = knowledge_cache.key_cache[0].shape[-2]
    print(f"KVcache prepared in {prepare_time} seconds")
    return knowledge_cache, prepare_time, kv_len

Iterate through each chunk to create the KV Cache

dynamic_cache_dict = {}

for i, (chunk, content) in enumerate(chunks_dictionary.items()):
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()
    gc.collect()
    print("*********")
    print(f"iteration - {i}")
    print("token length: ", len(content.split()))
    knowledge_cache, prepare_time, kv_len = dynamic_cache_creator(content, chunk)

print("KV cache generated successfully.")


############################Response#######################################
*********
iteration - 0
token length:  313
Before Embedding Step:
Memory Allocated: 3208.00146484375, Memory Reserved: 3310.0
After Caching Step:
Memory Allocated: 3362.84375, Memory Reserved: 3490.0
After Deletion of Everything Step:
Memory Allocated: 3210.43505859375, Memory Reserved: 3314.0
kvlen:  623
KVcache prepared in 0.31918978691101074 seconds
*********
iteration - 1
token length:  251
Before Embedding Step:
Memory Allocated: 3208.60693359375, Memory Reserved: 3310.0
After Caching Step:
Memory Allocated: 3329.39892578125, Memory Reserved: 3452.0
After Deletion of Everything Step:
Memory Allocated: 3210.50537109375, Memory Reserved: 3310.0
kvlen:  486
KVcache prepared in 0.29976963996887207 seconds
*********
iteration - 2
token length:  48
Before Embedding Step:
Memory Allocated: 3208.07177734375, Memory Reserved: 3310.0
After Caching Step:
Memory Allocated: 3249.330078125, Memory Reserved: 3356.0
After Deletion of Everything Step:
Memory Allocated: 3208.72021484375, Memory Reserved: 3310.0
kvlen:  166
KVcache prepared in 0.232680082321167 seconds
*********
iteration - 3
token length:  343
Before Embedding Step:
Memory Allocated: 3206.82177734375, Memory Reserved: 3310.0
After Caching Step:
Memory Allocated: 3379.5302734375, Memory Reserved: 3496.0
After Deletion of Everything Step:
Memory Allocated: 3209.52490234375, Memory Reserved: 3312.0
kvlen:  692
KVcache prepared in 0.4179873466491699 seconds
*********
iteration - 4
token length:  181
Before Embedding Step:
Memory Allocated: 3208.87646484375, Memory Reserved: 3312.0
After Caching Step:
Memory Allocated: 3307.05126953125, Memory Reserved: 3410.0
After Deletion of Everything Step:
Memory Allocated: 3210.41943359375, Memory Reserved: 3312.0
kvlen:  395
KVcache prepared in 0.2764251232147217 seconds
*********
iteration - 5
token length:  15
Before Embedding Step:
Memory Allocated: 3207.71630859375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3232.322265625, Memory Reserved: 3336.0
After Deletion of Everything Step:
Memory Allocated: 3208.10302734375, Memory Reserved: 3308.0
kvlen:  99
KVcache prepared in 0.24182796478271484 seconds
*********
iteration - 6
token length:  597
Before Embedding Step:
Memory Allocated: 3206.56005859375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3484.63916015625, Memory Reserved: 3654.0
After Deletion of Everything Step:
Memory Allocated: 3214.56005859375, Memory Reserved: 3308.0
kvlen:  1104
KVcache prepared in 0.49320268630981445 seconds
*********
iteration - 7
token length:  79
Before Embedding Step:
Memory Allocated: 3214.17333984375, Memory Reserved: 3306.0
After Caching Step:
Memory Allocated: 3271.05810546875, Memory Reserved: 3370.0
After Deletion of Everything Step:
Memory Allocated: 3215.05615234375, Memory Reserved: 3308.0
kvlen:  226
KVcache prepared in 0.25739097595214844 seconds
*********
iteration - 8
token length:  15
Before Embedding Step:
Memory Allocated: 3207.05615234375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3231.662109375, Memory Reserved: 3336.0
After Deletion of Everything Step:
Memory Allocated: 3207.44287109375, Memory Reserved: 3308.0
kvlen:  99
KVcache prepared in 0.23113107681274414 seconds
*********
iteration - 9
token length:  584
Before Embedding Step:
Memory Allocated: 3206.56005859375, Memory Reserved: 3306.0
After Caching Step:
Memory Allocated: 3490.56884765625, Memory Reserved: 3656.0
After Deletion of Everything Step:
Memory Allocated: 3214.56005859375, Memory Reserved: 3306.0
kvlen:  1128
KVcache prepared in 0.4444541931152344 seconds
*********
iteration - 10
token length:  15
Before Embedding Step:
Memory Allocated: 3214.17333984375, Memory Reserved: 3306.0
After Caching Step:
Memory Allocated: 3238.779296875, Memory Reserved: 3334.0
After Deletion of Everything Step:
Memory Allocated: 3214.56005859375, Memory Reserved: 3306.0
kvlen:  99
KVcache prepared in 0.22584986686706543 seconds
*********
iteration - 11
token length:  662
Before Embedding Step:
Memory Allocated: 3206.56005859375, Memory Reserved: 3306.0
After Caching Step:
Memory Allocated: 3516.56982421875, Memory Reserved: 3688.0
After Deletion of Everything Step:
Memory Allocated: 3214.56005859375, Memory Reserved: 3306.0
kvlen:  1233
KVcache prepared in 0.4652903079986572 seconds
*********
iteration - 12
token length:  15
Before Embedding Step:
Memory Allocated: 3214.17333984375, Memory Reserved: 3306.0
After Caching Step:
Memory Allocated: 3238.779296875, Memory Reserved: 3334.0
After Deletion of Everything Step:
Memory Allocated: 3214.56005859375, Memory Reserved: 3306.0
kvlen:  99
KVcache prepared in 0.25168299674987793 seconds
*********
iteration - 13
token length:  544
Before Embedding Step:
Memory Allocated: 3206.56005859375, Memory Reserved: 3306.0
After Caching Step:
Memory Allocated: 3499.1298828125, Memory Reserved: 3672.0
After Deletion of Everything Step:
Memory Allocated: 3214.37255859375, Memory Reserved: 3326.0
kvlen:  1164
KVcache prepared in 0.41371703147888184 seconds
*********
iteration - 14
token length:  15
Before Embedding Step:
Memory Allocated: 3213.98583984375, Memory Reserved: 3326.0
After Caching Step:
Memory Allocated: 3238.591796875, Memory Reserved: 3354.0
After Deletion of Everything Step:
Memory Allocated: 3214.37255859375, Memory Reserved: 3326.0
kvlen:  99
KVcache prepared in 0.23278450965881348 seconds
*********
iteration - 15
token length:  411
Before Embedding Step:
Memory Allocated: 3206.56005859375, Memory Reserved: 3306.0
After Caching Step:
Memory Allocated: 3411.7890625, Memory Reserved: 3544.0
After Deletion of Everything Step:
Memory Allocated: 3209.78271484375, Memory Reserved: 3312.0
kvlen:  825
KVcache prepared in 0.401630163192749 seconds
*********
iteration - 16
token length:  77
Before Embedding Step:
Memory Allocated: 3209.39599609375, Memory Reserved: 3312.0
After Caching Step:
Memory Allocated: 3272.7744140625, Memory Reserved: 3382.0
After Deletion of Everything Step:
Memory Allocated: 3210.39208984375, Memory Reserved: 3312.0
kvlen:  255
KVcache prepared in 0.24164104461669922 seconds
*********
iteration - 17
token length:  115
Before Embedding Step:
Memory Allocated: 3207.16943359375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3293.16552734375, Memory Reserved: 3394.0
After Deletion of Everything Step:
Memory Allocated: 3208.52099609375, Memory Reserved: 3308.0
kvlen:  346
KVcache prepared in 0.26290369033813477 seconds
*********
iteration - 18
token length:  136
Before Embedding Step:
Memory Allocated: 3207.52490234375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3313.1767578125, Memory Reserved: 3414.0
After Deletion of Everything Step:
Memory Allocated: 3209.17333984375, Memory Reserved: 3310.0
kvlen:  422
KVcache prepared in 0.3294835090637207 seconds
*********
iteration - 19
token length:  115
Before Embedding Step:
Memory Allocated: 3207.82177734375, Memory Reserved: 3310.0
After Caching Step:
Memory Allocated: 3295.5576171875, Memory Reserved: 3398.0
After Deletion of Everything Step:
Memory Allocated: 3209.20068359375, Memory Reserved: 3310.0
kvlen:  353
KVcache prepared in 0.29401135444641113 seconds
*********
iteration - 20
token length:  17
Before Embedding Step:
Memory Allocated: 3207.55224609375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3236.38330078125, Memory Reserved: 3340.0
After Deletion of Everything Step:
Memory Allocated: 3208.00537109375, Memory Reserved: 3308.0
kvlen:  116
KVcache prepared in 0.24158120155334473 seconds
*********
iteration - 21
token length:  29
Before Embedding Step:
Memory Allocated: 3206.62646484375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3241.1630859375, Memory Reserved: 3346.0
After Deletion of Everything Step:
Memory Allocated: 3207.16162109375, Memory Reserved: 3308.0
kvlen:  137
KVcache prepared in 0.23287725448608398 seconds
*********
iteration - 22
token length:  65
Before Embedding Step:
Memory Allocated: 3206.70849609375, Memory Reserved: 3306.0
After Caching Step:
Memory Allocated: 3261.55810546875, Memory Reserved: 3368.0
After Deletion of Everything Step:
Memory Allocated: 3207.55615234375, Memory Reserved: 3308.0
kvlen:  217
KVcache prepared in 0.22940826416015625 seconds
*********
iteration - 23
token length:  59
Before Embedding Step:
Memory Allocated: 3207.02099609375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3263.90576171875, Memory Reserved: 3370.0
After Deletion of Everything Step:
Memory Allocated: 3207.90380859375, Memory Reserved: 3310.0
kvlen:  226
KVcache prepared in 0.23958396911621094 seconds
*********
iteration - 24
token length:  103
Before Embedding Step:
Memory Allocated: 3207.05615234375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3290.36376953125, Memory Reserved: 3390.0
After Deletion of Everything Step:
Memory Allocated: 3208.36083984375, Memory Reserved: 3308.0
kvlen:  334
KVcache prepared in 0.2552649974822998 seconds
*********
iteration - 25
token length:  250
Before Embedding Step:
Memory Allocated: 3207.47802734375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3374.0927734375, Memory Reserved: 3488.0
After Deletion of Everything Step:
Memory Allocated: 3210.08740234375, Memory Reserved: 3312.0
kvlen:  668
KVcache prepared in 0.3109438419342041 seconds
*********
iteration - 26
token length:  22
Before Embedding Step:
Memory Allocated: 3208.78271484375, Memory Reserved: 3312.0
After Caching Step:
Memory Allocated: 3240.09912109375, Memory Reserved: 3346.0
After Deletion of Everything Step:
Memory Allocated: 3209.27490234375, Memory Reserved: 3312.0
kvlen:  126
KVcache prepared in 0.2368924617767334 seconds
*********
iteration - 27
token length:  26
Before Embedding Step:
Memory Allocated: 3206.66552734375, Memory Reserved: 3310.0
After Caching Step:
Memory Allocated: 3239.4736328125, Memory Reserved: 3344.0
After Deletion of Everything Step:
Memory Allocated: 3207.18115234375, Memory Reserved: 3310.0
kvlen:  132
KVcache prepared in 0.2281181812286377 seconds
*********
iteration - 28
token length:  19
Before Embedding Step:
Memory Allocated: 3206.68896484375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3237.259765625, Memory Reserved: 3342.0
After Deletion of Everything Step:
Memory Allocated: 3207.16943359375, Memory Reserved: 3308.0
kvlen:  123
KVcache prepared in 0.2610602378845215 seconds
*********
iteration - 29
token length:  74
Before Embedding Step:
Memory Allocated: 3206.65380859375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3270.0322265625, Memory Reserved: 3380.0
After Deletion of Everything Step:
Memory Allocated: 3207.64990234375, Memory Reserved: 3308.0
kvlen:  255
KVcache prepared in 0.27851366996765137 seconds
*********
iteration - 30
token length:  121
Before Embedding Step:
Memory Allocated: 3207.16943359375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3303.35595703125, Memory Reserved: 3406.0
After Deletion of Everything Step:
Memory Allocated: 3208.68115234375, Memory Reserved: 3308.0
kvlen:  387
KVcache prepared in 0.2641787528991699 seconds
*********
iteration - 31
token length:  23
Before Embedding Step:
Memory Allocated: 3207.68505859375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3239.00146484375, Memory Reserved: 3344.0
After Deletion of Everything Step:
Memory Allocated: 3208.17724609375, Memory Reserved: 3308.0
kvlen:  126
KVcache prepared in 0.2626781463623047 seconds
*********
iteration - 32
token length:  141
Before Embedding Step:
Memory Allocated: 3206.66552734375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3318.4111328125, Memory Reserved: 3440.0
After Deletion of Everything Step:
Memory Allocated: 3208.40771484375, Memory Reserved: 3308.0
kvlen:  446
KVcache prepared in 0.27580976486206055 seconds
*********
iteration - 33
token length:  20
Before Embedding Step:
Memory Allocated: 3207.91552734375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3240.41259765625, Memory Reserved: 3344.0
After Deletion of Everything Step:
Memory Allocated: 3208.41162109375, Memory Reserved: 3308.0
kvlen:  127
KVcache prepared in 0.27104806900024414 seconds
*********
iteration - 34
token length:  95
Before Embedding Step:
Memory Allocated: 3206.66943359375, Memory Reserved: 3308.0
After Caching Step:
Memory Allocated: 3282.2265625, Memory Reserved: 3384.0
After Deletion of Everything Step:
Memory Allocated: 3207.85693359375, Memory Reserved: 3308.0
kvlen:  304
KVcache prepared in 0.2703537940979004 seconds
KV cache generated successfully.

Load the Chunk Summaries

summary_dictionary = load_json_data("./summary.json")

Create a Vector Store and load the chunk summaries

from langchain_community.vectorstores  import Chroma
embeddings = HuggingFaceEmbeddings()

def create_and_initialize_retriever(summary_dict):
    id_key = "doc_id"
    doc_ids = list(summary_dict.keys())
    summary_texts = [Document(page_content=s, metadata={id_key: doc_ids[i]}) for i, s in enumerate(summary_dict.values())]
    vectorstore = Chroma.from_documents(documents=summary_texts, embedding=embeddings)

    return vectorstore


vectorstore = create_and_initialize_retriever(summary_dictionary)
#
print(len(vectorstore.get()['documents'])
##############################Response#############################
35

Generate Answers

Here we have applied three approaches

1. RAG only
2. CAG only
3. Vectorstore Chunk + CAG

Approach 1 RAG:

from langchain_core.prompts import PromptTemplate
prompt = """ Based on the context provided below please provide an answer to the question asked.
If the context provided is not sufficient to answer the question then do not make up your own answer.Instaed reply with "No sufficient information available".
CONTEXT
===============================================
{content}
===============================================
QUESTION
===============================================
{question}
================================================
ANSWER
"""
query = "What is Fibromyalgia?"
contexts = vectorstore.similarity_search(query)
final_prompt = PromptTemplate(template=prompt,input_variables=["context","question"],)
#
chain = final_prompt | llm
#
response = chain.invoke({"content":[d.page_content for d in contexts],"question":query})
#
print(response.content)

##########################Response##############################
'Fibromyalgia is a chronic disorder characterized by widespread pain and tenderness. It affects 2-4% of the population and is more common in women. It is often accompanied by other symptoms such as fatigue, mood issues, and sleep disturbances. Fibromyalgia can co-occur with other chronic pain conditions, psychiatric disorders, and restless legs syndrome. The American College of Rheumatology (ACR) has developed diagnostic criteria for fibromyalgia, with updates in 2010, 2011, and 2016. The 2019 AAPT diagnostic criteria are also used. Fibromyalgia should be considered in patients with rheumatologic diagnoses who do not respond well to treatment of their primary condition.'

Approach 2 : CAG : Use KV cache to generate answer embeddings

def clean_up(cache:DynamicCache,origin_len:int):
    for i in range(len(cache.key_cache)):
        cache.key_cache[i] = cache.key_cache[i][:,:,:origin_len,:]
        cache.value_cache[i] = cache.value_cache[i][:,:,:origin_len,:]
#
# Query we have already defined in the extraction of embeddings phase
generation_prompt = f"""
    {query}
    Give very concise answer. In max one sentence
    <|eot_id|>
    <|start_header_id|>assistant<|end_header_id|>
    """
#
def generate(
    model,
    input_ids: torch.Tensor,
    past_key_values,
    max_new_tokens: int = 300
) -> torch.Tensor:
    """
    Generate text with greedy decoding.

    Args:
        model: HuggingFace model with automatic device mapping
        input_ids: Input token ids
        past_key_values: KV Cache for knowledge
        max_new_tokens: Maximum new tokens to generate
    """

    embed_device = model.model.embed_tokens.weight.device

    origin_ids = input_ids
    input_ids = input_ids.to(embed_device)

    output_ids = input_ids.clone()
    next_token = input_ids

    with torch.no_grad():
        for _ in range(max_new_tokens):
            outputs = model(
                input_ids=next_token, 
                past_key_values=past_key_values,
                use_cache=True
            )
            next_token_logits = outputs.logits[:, -1, :]
            next_token = next_token_logits.argmax(dim=-1).unsqueeze(-1)
            next_token = next_token.to(embed_device)

            past_key_values = outputs.past_key_values

            output_ids = torch.cat([output_ids, next_token], dim=1)

            if next_token.item() in model.config.eos_token_id:
                break
    return output_ids[:, origin_ids.shape[-1]:]

Helper Function to generate answers from KV cache

def generate_answer(kv_cache,query,model,tokenizer,origin_len):
    clean_up(kv_cache,origin_len)
    device = model.model.embed_tokens.weight.device
    input_ids = tokenizer.encode(query+"\n", return_tensors="pt").to(model.device)
    answer_embeddings = generate(model, input_ids, kv_cache)
    generated_text = tokenizer.decode(answer_embeddings[0], skip_special_tokens=True, temperature=0.5)
    print(generated_text)
    return generated_text

Generate answer from KV Cache

query = "What is Fibromyalgia?"
# Query we have already defined in the extraction of embeddings phase
generation_prompt = f"""
    {query}
    Give very concise answer. In max one sentence
    <|eot_id|>
    <|start_header_id|>assistant<|end_header_id|>
    """
response = generate_answer(knowledge_cache,generation_prompt,model,tokenizer,origin_len)

##########################Response###########################
Fibromyalgia is a chronic pain condition characterized by widespread musculoskeletal pain, fatigue, and other somatic symptoms, affecting 10-15% of adults worldwide, with women being diagnosed more frequently than men.

Approach 3 Vectorstore Chunk + CAG

#Fetch Correct Chunks
def fetch_correct_chunk(query, vectorstore):

    # embedding_vector = embeddings.embed_query(query)
    # docs = vectorstore.similarity_search_by_vector(embedding_vector)
    docs = vectorstore.similarity_search(query)

    chunk = docs[0].metadata["doc_id"]
    for d in docs:
        print(d.metadata)
    print(docs[0].page_content)
    return chunk
#Extract the correct k-v cache
#
def initialize_knowledge_cache(chunk):
    knowledge_cache = OffloadedCache()
    knowledge_cache.key_cache = torch.load(f"./chunk_caches/{chunk}_key.pt", weights_only=False)
    knowledge_cache.value_cache = torch.load(f"./chunk_caches/{chunk}_value.pt", weights_only=False)
    knowledge_cache.prefetch_stream = torch.cuda.Stream()
    knowledge_cache.original_device = torch.load(f"./chunk_caches/{chunk}_od.pt", weights_only=False)
    return knowledge_cache
#
knowledge_cache = initialize_knowledge_cache(chunk_name)
#
# Query we have already defined in the extraction of embeddings phase
generation_prompt = f"""
    {query}
    Give very concise answer. In max one sentence
    <|eot_id|>
    <|start_header_id|>assistant<|end_header_id|>
    """
#
chunk_name = fetch_correct_chunk(query, vectorstore)
chunk_name = f"chunk_{int(chunk_name.split('_')[-1])+1}"
#
input_ids = tokenizer.encode(generation_prompt, return_tensors="pt").to(model.device)
answer_embeddings = generate(model, input_ids, knowledge_cache)
#
generated_text = tokenizer.decode(answer_embeddings[0], skip_special_tokens=True, temperature=0.5)
print(generated_text)
###########################Response#####################################
 Fibromyalgia is a chronic pain condition characterized by widespread musculoskeletal pain, fatigue, and other somatic symptoms, affecting 10-15% of adults worldwide, with women being diagnosed more frequently than men

New Query

query = "What are the causes of Fibromyalgia?"

Response from CAG

# Query we have already defined in the extraction of embeddings phase
generation_prompt = f"""
    {query}
    Give very concise answer. In max one sentence
    <|eot_id|>
    <|start_header_id|>assistant<|end_header_id|>
    """
response = generate_answer(knowledge_cache,generation_prompt,model,tokenizer,origin_len)

#####################Response#####################################
Fibromyalgia is a clinical diagnosis, and the causes include: 
    • Fatigue, 
    • Pain in the neck, back, and other areas of the body 
    • Tension headaches 
    • Sleep disturbances 
    • Mood changes 
    • Cognitive difficulties 
    • Depression 
    • Anxiety 
    • Irritability 
    • Sleep apnea 
    • Hypothyroidism 
    • Hyperthyroidism 
    • Hyperparathyroidism 
    • Neurological disorders

Response from RAG

contexts = vectorstore.similarity_search(query)
chain = final_prompt | llm
response = chain.invoke({"content":[d.page_content for d in contexts],"question":query})
print(response.content)
############################Response#########################
'The causes of fibromyalgia are likely to include disordered central nociceptive signal processing, with suggested etiologies such as dysfunction of the hypothalamic-pituitary-adrenal axis, autonomic nervous system, diffuse inflammation, glial cell activation, small fiber neuropathy, and infections. Additionally, twin studies suggest a genetic component may also be a factor.'

Answer Query

query = "Do people suffering from rheumatologic conditions may have fibromyalgia?"

Response from CAG

generation_prompt = f"""
    {query}
    Give very concise answer. In max one sentence
    <|eot_id|>
    <|start_header_id|>assistant<|end_header_id|>
    """
#
response = generate_answer(knowledge_cache,generation_prompt,model,tokenizer,origin_len)
############################Response#####################
People with rheumatologic conditions, such as rheumatoid arthritis, lupus, or rheumatoid factor-positive anemia, may have fibromyalgia.

Response from RAG

contexts = vectorstore.similarity_search(query)
chain = final_prompt | llm
response = chain.invoke({"content":[d.page_content for d in contexts],"question":query})
print(response.content)\
#############################Response################################
'Yes, fibromyalgia should be considered in patients with rheumatologic diagnoses who do not respond well to treatment of their primary condition.'

Answer Query

query = "Mention the nonpharmacologic treatment for fibromyalgia?"

Response from CAG

generation_prompt = f"""
    {query}
    Give very concise answer. In max one sentence
    <|eot_id|>
    <|start_header_id|>assistant<|end_header_id|>
    """
response = generate_answer(knowledge_cache,generation_prompt,model,tokenizer,origin_len)

##################################Response###############################
 Nonpharmacologic treatment for fibromyalgia includes:
    • Cognitive behavioral therapy (CBT) and cognitive behavioral therapy for insomnia (CBT-I)
    • Physical therapy
    • Exercise and relaxation techniques
    Nonpharmacologic treatment for fibromyalgia includes:
    • Cognitive behavioral therapy (CBT) and cognitive behavioral therapy for insomnia (CBT-I)
    • Physical therapy
    • Exercise and relaxation techniques

Response from RAG

contexts = vectorstore.similarity_search(query)
chain = final_prompt | llm
response = chain.invoke({"content":[d.page_content for d in contexts],"question":query})
print(response.content)

##################################Response#############################
'Nonpharmacologic treatments for fibromyalgia include patient education, treatment of comorbid conditions, lifestyle modification, cognitive behavior therapy (CBT), and self-management support. Exercise, particularly aerobic exercise of moderate intensity, has the strongest evidence for improving pain, function, fatigue, and sleep quality. CBT shows moderate-quality evidence for modest pain and disability improvements. Complementary and alternative medicine options like acupuncture, massage, meditation, and nutritional supplements have limited high-quality evidence but may help. Yoga, Pilates, and tai chi have shown promise in reducing pain and improving function. Manual therapy, specifically myofascial release, may decrease symptoms.'

Answer Query

query = "What are the medications and doses for Fibromyalgia?"

Response from CAG

generation_prompt = f"""
    {query}
    Give very concise answer. In max one sentence
    <|eot_id|>
    <|start_header_id|>assistant<|end_header_id|>
    """
response = generate_answer(knowledge_cache,generation_prompt,model,tokenizer,origin_len)

#################################Response###########################
 The medications for fibromyalgia include: 
- Amitriptyline (Elavil) 25-100 mg/day 
- Duloxetine (Cymbalta) 10-60 mg/day 
- Gabapentin (Neurontin) 300-600 mg/day 
- Pregabalin (Lyrica) 150-300 mg/day

Response from RAG

contexts = vectorstore.similarity_search(query)
chain = final_prompt | llm
response = chain.invoke({"content":[d.page_content for d in contexts],"question":query})
print(response.content)

#############################Response #######################################
The article does not provide specific dosages for the medications used to treat fibromyalgia. It recommends starting a single medication at a low dosage, gradually increasing to the recommended dosage, and continuing for at least three months to ensure an adequate trial. If a satisfactory clinical response is achieved, treatment should be continued for at least 12 months. If there is no response, clinicians should assess medication adherence, maximize nonpharmacologic management, and exclude other conditions that may need additional intervention.

Three classes of medications are potentially useful: tricyclic antidepressants, serotonin-norepinephrine reuptake inhibitors, and gabapentinoids. Duloxetine, milnacipran, and pregabalin are approved by the U.S. Food and Drug Administration for the treatment of fibromyalgia. Tricyclic antidepressants, such as amitriptyline, can improve several symptoms of fibromyalgia, including pain, sleep, and patient satisfaction. Cyclobenzaprine, a muscle relaxant that is a tricyclic derivative, can be a reasonable option for patients unable to tolerate amitriptyline. There is insufficient evidence for other muscle relaxants in the treatment of fibromyalgia. Duloxetine and milnacipran have low-quality evidence for pain relief. Pregabalin has been shown to reduce pain, fatigue, and improve sleep and quality of life.

A systematic review by Choy et al. (2011) found that pregabalin, duloxetine, and milnacipran were more effective than placebo in improving pain and physical function in patients with fibromyalgia. White et al. (2018) analyzed real-world dosing patterns for the three FDA-approved medications for fibromyalgia: duloxetine, milnacipran, and pregabalin, and found that the majority of patients were prescribed doses lower than the recommended doses in the FDA-approved labeling. Nishishinya et al. (2008) found that amitriptyline was more effective than placebo in improving pain and sleep disturbance in patients with fibromyalgia. Tofferi et al. (2004) found that cyclobenzaprine was more effective than placebo in improving pain and sleep disturbance in patients with fibromyalgia. Welsch et al. (2018) found that duloxetine and milnacipran were more effective than placebo in improving pain and physical function in patients with fibromyalgia. However, it is important to note that the optimal dosing and duration of these medications may vary among individuals, and real-world dosing patterns may differ from those recommended in the FDA-approved labeling.

Answer Query

query = "What is the starting dosage of Amitriptyline?"

Response from CAG

generation_prompt = f"""
    {query}
    Give very concise answer. In max one sentence
    <|eot_id|>
    <|start_header_id|>assistant<|end_header_id|>
    """
response = generate_answer(knowledge_cache,generation_prompt,model,tokenizer,origin_len)

###################################Response #############################
Amitriptyline is typically started at a dose of 25-50 mg daily.

Response from RAG

contexts = vectorstore.similarity_search(query)
chain = final_prompt | llm
response = chain.invoke({"content":[d.page_content for d in contexts],"question":query})
print(response.content)

########################################Response######################
The text does not provide information on the starting dosage of Amitriptyline for the treatment of fibromyalgia. It only mentions that Amitriptyline can improve several symptoms of fibromyalgia, including pain, sleep, and patient satisfaction. Therefore, the answer is "No sufficient information available"

From the above experiment it is evident that there arescenarios where CAG has outperformed RAG and vice versa.

So RAG or CAG ?

RAG: The Dynamic Retriever 📡

In a RAG setup, the system employs both dense and sparse retrieval techniques to fetch the most relevant documents. These documents are then combined with the user’s query and fed into the model for processing. It’s like a research assistant quickly gathering information to help answer a question.

🧑‍🎓CAG: The Preloaded Knowledge Base 📂

CAG, on the other hand, takes a different approach. It preloads all relevant information into the model’s KV cache. When a query is received, the model processes it directly using this pre-existing, readily available knowledge base. Think of it as having all the answers at your fingertips! 💡Key Performance

Takeaways: 🔑

• Accuracy: CAG’s Edge ✅: CAG can outperform RAG, particularly when the entire relevant knowledge base can fit within the model’s context window. Moreover, CAG offers highly consistent results for tasks involving stable, unchanging knowledge. Essentially, when it can hold all the cards, CAG plays them perfectly. 🃏
• Latency & Complexity: CAG’s Speed and Simplicity ⚡: RAG often suffers from added latency due to the time-consuming retrieval step. CAG bypasses this entirely, resulting in faster response times and a more efficient process. By eliminating the need to search, CAG streamlines the entire operation. 🚀

🔍🌐✍️Optimal Use Cases for RAG (Retrieval-Augmented Generation):
✅ Dynamic, Time-Sensitive Domains
Examples: Financial markets, live news reporting, or medical research.
RAG shines when information evolves rapidly, enabling real-time access to external databases for up-to-the-minute accuracy.

✅ Large-Scale Enterprise Knowledge Bases
Examples: Corporate documentation, technical repositories, or customer service logs.
RAG efficiently handles datasets exceeding LLM context windows, leveraging scalable retrieval systems to surface relevant data on demand.

✅ Audit-Critical Applications
Examples: Legal analysis, academic writing, or compliance reporting.
RAG provides source citations by design, ensuring traceability and verifiability of generated responses.

🧠🗄️⚡Optimal Use Cases for CAG (Context-Augmented Generation):
✅ Controlled Knowledge Environments
Examples: Static reference materials, product specifications, or internal policy guidelines.
CAG operates efficiently with compact, stable datasets that fit entirely within the model’s memory.

✅ High-Performance Conversational Systems
Examples: Chatbots, virtual assistants, or FAQ engines.
CAG prioritizes speed and consistency for repetitive queries, avoiding latency from external data retrieval.

✅ Resource-Constrained Implementations
Examples: Edge computing, lightweight applications, or rapid prototyping.
CAG simplifies architecture by eliminating infrastructure overhead for database management and indexing.

Combined Workflow (RAG vs. CAG):

CAG: 🧠🗄️⚡ (Brain + Cache + Speed) →Fixed knowledge, fast responses.
RAG: 🔍🌐✍️ (Search + Global Data + Writing) →Dynamic retrieval, context-aware generation.

Key Differentiation:
RAG dynamically bridges external knowledge gaps, while CAG optimizes for speed and simplicity within fixed contexts.

Combining CAG and RAG

A combined method offers the best of both worlds 🌎: using CAG for frequently accessed documents and RAG for rare or new information. 🏗️ It’s like having a well-stocked library 📚 with a super-fast search engine 🔍!

✔️ Flexibility 🤸: This approach lets you maintain speed and simplicity with CAG, while RAG handles the dynamic aspects. Adapt to any situation! 🔄

✔️ Efficiency ⚙️: By combining both methods, you avoid the complexity of a full RAG pipeline, ensuring streamlined performance. Get the job done faster and easier! 🚀

Conclusion: CAG vs. RAG — Choosing the Right Tool

🔍 No One-Size-Fits-All Solution
The choice depends on your data’s scale, volatility, and performance needs:

Retrieval-Augmented Generation (RAG) 🌐📈

Strengths:

• Handles massive, ever-changing datasets (e.g., live news, global markets).
• Delivers traceable, up-to-date answers with source citations.
• Ideal for: Dynamic environments requiring accuracy at scale.

Cache-Augmented Generation (CAG) 🗄️⚡

Strengths:

• Optimizes speed and simplicity for stable, compact knowledge bases.
• Reduces latency and infrastructure complexity.
• Ideal for: Repetitive queries in controlled, resource-limited settings.

Future Outlook 🔮🚀

As LLMs evolve with larger context windows and long-term memory, CAG will gain traction for medium-sized datasets. However, RAG remains indispensable for real-time, large-scale applications.

Final Takeaway ⚖️✨

• Prioritize RAG when scale and freshness matter.
• Choose CAG for speed and simplicity in stable contexts.
• Hybrid approaches may bridge gaps as both technologies advance.

By aligning our strategy with these strengths, we can balance accuracy, efficiency, and scalability in AI-driven systems. 🌟

References

• Don’t Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
• https://pub.towardsai.net/retrieval-augmented-generation-rag-vs-0b4bc63c1653

connect with me