Harry Potter and the Self-Learning Knowledge Graph RAG
With this demo, we wanted to demonstrate 3 key powerful applications of knowledge graph in RAG, and demonstrate how it can improve RAG accuracy, dramatically reduce time to production, and is one of the first examples of a self-learning RAG.
- Recursive Retrieval
- On-demand Ontologies/ Automated Knowledge Graph
- Memory and Multi-Hop Reasoning
In this demo, we upload chapters of the Harry Potter book and see how an automated knowledge graph is constructed as new chapters are added, with the knowledge graph detecting and capturing the relationships and entities relevant to the specific question at hand, as opposed to capturing all relationships and entities at random levels of granularity. In this format, it can be set to recursively retrieve and iteratively construct knowledge graphs over time and each time new information is introduced.
We then demonstrate how the knowledge graph can automatically conjoin and merge based on all the mini-knowledge graphs created each time a separate question is asked, allowing it to learn over time, each time a question is asked. This allows for the creation of relevant knowledge bases / second brains, simply by throwing a laundry list of questions at this automated knowledge graph tool. Besides simplifying development processes, this knowledge base acts as a memory tool for the LLM and easily performs multi-hop reasoning.
For our tech stack, we used Langchain, Pinecone Serverless, OpenAI.
Recursive retrieval
Knowledge graphs can act as a context store, fixing and repeatedly retrieving information based around a single concept.
Since this mini-knowledge graph is of a single concept, retrieval is thus more scoped in and structured to the question which contained the concept. This allows for retrieval of data related to a single concept across multiple documents over time, as well as over different chunks.
Regarding retrieval over time, imagine your RAG system constantly having new updates to the vector database and new information flows in that you want to append to a historically existing answer. Now you can.
Regarding retrieval over different chunks, imagine information related to the core concept of the question is spread across many different documents or pages, and you want to make sure all is retrieved and structured automatically. Now you can.
The knowledge graph acts as a contextually aware filter that iteratively and constantly adds information to the relevant concepts at focus.
On-Demand Knowledge Graph
At a conceptual level, a Knowledge Graph is a straightforward thing. It creates deterministic relationships between concepts, so you can understand how things are related to each other.
The biggest problem that people new to building with knowledge graphs face is in understanding what is the right ontology to be building for the right context.
An ontology is essentially the concepts and the relationships that you want to track within the graph. In the above graph, the concept “Author”, and the relationship “hasAuthor” are all part of the ontology, that is hopefully useful for the specific context the creator had in mind.
With LLMs and context of the specific question that is being asked, the ontology that is to be created may wildly vary. Even within the same domain, different questions will have different ontologies.
In the demo above, we built a workflow that allows for LLM-driven context-aware knowledge graph creation. Depending on the question being asked, the knowledge graph only captures the concepts and relationships that are specifically relevant to the question. This is relevant to our belief that capturing all types of relationships and entities in a large knowledge graph is inefficient compared to focusing on capturing the relationships and entities that are context and question specific.
The key step here that will accelerate knowledge graph adoption is to include workflow tools to allow human-in-the-loop management of the knowledge graph construction. Although we are currently pretty far from it, over time, we anticipate that the models themselves can automate more and more of the creation of relevant, mini-knowledge graphs, although the greater adoption of knowledge graphs and LLMs in general will increase the number of use-cases and granularity of control that is demanded and does not currently exist.
Memory and Multi-Hop Reasoning
With the Harry Potter demo, we can see that the first mini-knowledge graph created from the question, “What does Harry wear?” is stored as a contextual memory for concepts regarding Harry.
We can see that when a conceptually linked question, “What does Dumbledore wear?” is asked, the second mini-knowledge graph is created, detects potentially overlapping relationships, and is synced automatically to the first mini-knowledge graph.
This is a demonstration of memory within RAG pipelines, specifically of the Harry Potter mini-KG. When the last question is then asked, “What do Harry and Dumbledore wear?”, when the search is now performed, the LLM first checks the memory store for relevant relationships and facts. It detects that the Harry and Dumbledore conjoined graph is relevant for retrieval and retrieves relevant to ‘Harry’ & ‘Dumbledore’ & ‘has’. This is multi-hop reasoning performed in a single retrieval action conjoining and traversing the conceptual relationship that different entities share across different documents.
Memory is a key primitive necessary to enable LLMs to perform multi-hop reasoning across concepts from various different pages or documents.
One very interesting consequence of implementing knowledge graph systems in RAG pipelines also means that due to the intelligent nature of context gathering within the system, the RAG system can be tested in a manner familiar to ML engineers: by throwing 100s of synthetic questions at the RAG system and letting knowledge graph and context build automatically over time. This would be one of the first examples of how to create a self-learning RAG system.
Memory, Automated Knowledge Graphs, Multi-Hop Reasoning and Recursive Retrieval are all extremely key parts of an accurate, enterprise, advanced RAG system. These are how these advanced RAG techniques come together to create an increasingly smarter RAG system.
WhyHow.AI is building tools to help developers bring more determinism and control to their RAG pipelines using graph structures. If you’re thinking about, in the process of, or have already incorporated knowledge graphs in RAG, we’d love to chat at team@whyhow.ai, or follow our newsletter at WhyHow.AI. Join our discussions about rules, determinism and knowledge graphs in RAG on our newly-created Discord.
