5 Misconceptions of KG & RAG Systems — Building & Using RAG-Native Graphs
In this article, we wanted to tackle a few common misconceptions of knowledge graphs within RAG systems. We have especially been focused on understanding the difference between historical knowledge graph construction techniques, and nuances that come with what we call “RAG-Native Graphs”.
These misconceptions are:
Common Misconceptions #A: A large graph is needed
When people think about knowledge graphs, they sometimes get the idea that knowledge graphs are this large monolithic data structure that takes a lot of time and energy to create, spanning years and millions of dollars.
Historically, that has been true, for the specific use-cases that knowledge graphs have been used for pre-LLMs, such as large scale fraud detection or biomedical insights.
With LLMs, some of the use cases for knowledge graphs have changed, and LLMs are also used to automate a lot of knowledge graph construction, reducing work that used to take years into seconds.
Furthermore, with LLM systems, the opportunity for small graphs arises, further reducing the activation energy for creating valuable graphs.
Before LLMs, knowledge graphs were largely used akin to data dictionaries — a means to enforce semantic structure across different terms across different data silos, and to aggregate datasets to unveil hidden relationships, or to perform relationship mining. Knowledge graphs were used to capture concepts and relationships that would be interesting to track, especially against large unstructured data stores. An example of this would be collecting large bio-pharma academic papers and putting all the information into a knowledge graph to be able to understand hidden relationships between concepts across different papers, especially relationships that required tying relationships across multiple documents (i.e. Paper 1 says X = Y, Paper 2 says Y = Z. A knowledge graph can reveal that X = Z).
With the advanced development of LLMs, we should view knowledge graphs as tools for sharpening semantic focus and structural grounding , not just as aggregators of data. The resultant small graphs do not need to be fully complete because LLMs bring with them their own understanding of semantics. This means that, in many cases, it is not necessary to create a perfectly described version of the world.
Common Misconceptions #B: Knowledge Graphs are only useful for network-type data like payments or social media data
Knowledge graphs have historically been used a lot for network type data. Such data include payments or social-media type network data.
A common misconception is that knowledge graphs are then only relevant for network-based data, since that was traditionally the main focus of using KGs.
What has changed has been the prevalence of unstructured data. Just like how vector databases really took off in popularity when unstructured data search through using embeddings emerged, new paradigms of unstructured data storage and search are also present in different types of knowledge graphs that were not historically common.
Besides network-type data, hierarchical data types from unstructured data are also a great fit for knowledge graphs. I go into more detail about the types of hierarchical data that are present in this article, as well as give an example of a hierarchical knowledge graph, in the form of a representation of Apple’s quarterly financial report here.
In this graphic by Neo4J, they cover a non-exhaustive selection of the range of information that can be represented in graph structures. This reflects the many use-cases that graph structures can be relevant for unstructured data.
A concrete, real life example of non-network based data being represented in a graph structure for information retrieval can be seen in this customer service system built by LinkedIn.
Here, the knowledge graph is used to represent different elements of a range of customer support tickets, including the Priority Status, the Root Cause, the Description elements, and summary. This is a basic hierarchical representation of the elements of a customer support ticket, which is used to aid in the classification of a query (“Question Query”) and retrieval of specific elements of the ticket (“HAS_STEPS_TO_REPRODUCE”).
This hierarchical, non-network data structure allows for very precise retrieval of specific information on the basis of information stored in a pre-defined schema. We can also see overlap with Misconception #A with regards to a large graph, as not all the possible entities and relationships in the customer support ticket is represented in a graph, only specific aspects of the ticket that LinkedIn cares about are represented as edges and nodes.
We can see many other examples of KG RAG systems in the wild, including this one here about patient data, where multi-hop retrieval is not the value proposition being sought after, but rather deterministic and more complete information retrieval
Common Misconception #C: Clustering / Graph Analytics and Graph RAG overlap significantly
The process to create knowledge graphs has historically involved a range of methods, including clustering. We have found in our experience building KG RAG methodologies that clustering isn’t particularly relevant to RAG. This is because clustering is more about gaining a high-level understanding of the underlying data, whereas RAG is much more about targeted information retrieval, and the level of granularity offered by clustering simply is not enough to obtain precise results.
Clustering broadly refers to the idea that a model performs a series of categorizations on top of unstructured data, and classifies them into a number of categories that the model is fine-tuned to identify (i.e. “Information about People”, “Information about Jobs”, “Information about Companies”, etc). These categories form the basis of a schema, and the underlying information is put into a graph on the basis of this schema. Clustering is only a subset of techniques used for graph analytics. Some of the proposed arguments for why this is relevant to RAG are that this is a technique that can help to automate discovery of the potential schema to be used.
This technique is particularly helpful and useful when you are trying to gain a high-level understanding of the unstructured text, and to understand how the different themes of the text are related to each other. A lot of the value here is in the analytics of the nature of the underlying data.
However, as you can probably imagine, there are some glaring reasons why this is ill-suited for RAG processes. RAG is less about having high-level analytics of your unstructured data, but about representing your data granularly and precisely such that precise questions against the data can get precise results. Having a high-level categorization of your data, while potentially useful, does not clearly create more precise retrieval processes than a simple Vector RAG search. This is especially since the issue with RAG at the moment is not about a lack of a general retrieval, but overcoming issues with more precise and deterministic retrieval of relevant information.
While clustering may be a potential way to get a high-level schema to get started, it is unclear how it may outperform simply asking an LLM to give a high-level schema based on the underlying data, or any other techniques for schema generation. In our experience building KG RAG for our clients, we regularly observe that constructing a graph schema that conforms to a user’s business requirements and the type of questions being asked results in reasonably good performance, at the level of granularity typically seen in RAG
This is not to say clustering and graph analytics are not relevant. There are many instances where you can perform RAG and graph analytics across specific types of data, especially network-based data. For example, it may be desirable to create a knowledge graph at a level of granularity for the purposes of RAG and granular information retrieval, and then subsequently performing clustering analysis to gain a higher-level analysis of the underlying information.
Common Misconception #D: KG RAG is only useful for multi-hop queries and retrieval
In a similar vein to graph analytics, knowledge graphs have historically been useful for understanding hidden relationships between different data silos. This discovery process can be done through multi-hop retrieval, which makes it easy to retrieve information through multiple steps of inferences.
One core misconception is that knowledge graphs are only useful for multi-hop retrieval. Graph structures are useful for enforcing relationships and to represent hierarchical information as we discussed in Misconception #B. Sometimes these relationships take the form of multi-hop retrieval. However, in many cases in RAG, multi-hop retrieval is not the overwhelming problem being faced.
Many KG & RAG papers discuss KG as being especially relevant for RAG systems because of multi-hop queries. This distracts from the core value proposition of structured knowledge representation. We can see that many of these papers use various contortions to try and emphasize this specific value proposition in the relatively unnatural ways that questions are posed and benchmarked against.
Many of these queries used here as examples for the benefit of multi-hop queries tend to be a poor reflection of how people would typically frame questions.
We can see this sentiment reflected by practical KG & RAG systems in real-life use-cases like the LinkedIn example above. Real-life use cases such as the LinkedIn example above more strongly focus on the need for deterministic retrieval and association, rather than multi-hop reasoning, and this is far more representative of how enterprise RAG use cases are typically defined
That being said, there are many types of multi-hop queries, and multi-hop reasoning definitely has a role to play in KG & RAG systems. Multi-hop queries pop up in different systems and should be seen as a valuable additional benefit but not necessarily the core benefit. To overemphasize the role of multi-hop queries would be to distract from the core value of KG & RAG systems.
The value of KG & RAG systems should be emphasized in:
- Explainability
- Deterministic systems
- Completeness
We discuss these value propositions in more depth in our articles here:
- Explainability / Deterministic Systems: https://medium.com/enterprise-rag/understanding-the-knowledge-graph-rag-opportunity-694b61261a9c
- Completeness: https://medium.com/neo4j/graph-vs-vector-rag-benchmarking-optimization-levers-and-a-financial-analysis-example-001587219683
Common Misconception #E: Lower cost of processing and longer context windows reduces the need for KG RAG due to a large margin of error
KG RAG is not just about reducing the amount of irrelevant information for RAG. The biggest concern we run into is the ability to bring in relevant, but non-semantically similar information into the context window.
The LinkedIn example also happens to be a good example of this. Without the knowledge graph, it would be hard to associate the information in (“HAS_STEPS_TO_REPRODUCE”) and return it into the context window because those words do not appear in the answers itself.
Instead, knowledge graphs are great for creating relationships between the actual steps to reproduce the issue with the relationship (“HAS_STEPS_TO_REPRODUCE”). In other words, knowledge graphs are great for bringing in relevant but non-semantically similar information. Knowledge graphs shouldn’t be seen as a way to reduce costs of RAG, but as a way to bring in information that vector search is structurally unable to retrieve, increasing the determinism, accuracy and completeness of the answers provided.
WhyHow.AI is building tools to help developers bring more determinism and control to their RAG pipelines using graph structures. If you’re thinking about, in the process of, or have already incorporated knowledge graphs in RAG for accuracy, memory and determinism, we’d love to chat at team@whyhow.ai, or follow our newsletter at WhyHow.AI. Join our discussions about rules, determinism and knowledge graphs in RAG on our Discord.
