Build a RAG chatbot and optimize the performance through usage
RAG is widely used to leverage LLMs in practical applications like chatbots. In this project, a RAG system was implemented using publicly available datasets (GDPR dataset), with real-time monitoring to track usage and continuously improve its performance based on human feedback (explicit and implicit). OpenAI serves as the underlying LLM for the system.
The GitHub repository for the code can be found here. The real-time monitoring and optimization implementation is available here.
The process began by splitting the original PDF into 21 separate articles. Each article was then converted from PDF format into text files using a custom script (available in gdprqa/gdprqa/pdf_parsing/utils.py).
Once converted, semantic chunking was applied to break the content into meaningful sections. These chunks were vectorized and stored in ChromaDB, with each article assigned to its own collection for easier retrieval. Additionally, all chunks were combined into a single collection to facilitate cross-article searches when necessary.
Two helper models support the system: one identifies the most relevant article for answering a query, defaulting to a full-document search if no suitable article is found. The other model reformulates the query to enhance the effectiveness of semantic searches within the vector database.
You can ask questions about GDPR in the query section such as ‘How much does it cost to become GDPR compliant’, etc.
When you finish using the chatbot you want Feedback Intelligence (FI) to analyse the usage. Write exit to send the usage to FI.
To analyze this chatbot usage and optimize it, we use FI. For that, we created the GDPR chatbot project in the platform. Then through settings, generate the API key to connect the chatbot to the platform. As we can see the added and processed data is 0.
After exiting the chatbot the data has been sent to FI successfully and the data is 3 already (we did 3 queries). Insights Engine diagnoses the usage and provides insights on issues, topics, user satisfaction score, and more. The results are used to optimize the chatbot.
Drop us a line if you are building any RAG application.
co-authors Mels Hakobyan & Erik Harutyunyan
