Knowledge Graphs For Large Language Models

The potential impact of LLMs extends beyond language-related tasks. With their ability to process and comprehend vast amounts of information, LLMs can aid in decision-making processes, assist in scientific research, and contribute to problem-solving in complex domains like healthcare, finance, and engineering.

The recent advancements in LLMs have sparked excitement and enthusiasm in the AI community, as they bring us closer to realizing the vision of AGI. While there are still challenges to overcome, such as ethical considerations and the need for continuous improvement, the progress made so far is truly inspiring and holds tremendous promise for the future of AI.

Despite their success in many applications, LLMs have been criticized for their lack of factual knowledge. Particularly, LLMs memorise the knowledge and facts found in the training corpus. However, other research has shown that LLMs are unable to recollect facts and frequently experience hallucinations by coming up with false assertions (“Language models as knowledge bases?”, 2019).

This issue severely impairs the trustworthiness of LLMs.

LLMs represent knowledge implicitly in their parameters: This means that the model learns patterns and associations from the training data, and this knowledge is encoded within the model’s parameters.

Difficulty in interpreting or validating LLM knowledge: Since LLMs don’t explicitly store or present their knowledge, it can be hard to determine how accurate or reliable their conclusions or responses are.

LLMs use probability for reasoning: This probabilistic approach can sometimes lead to indecisive outcomes or uncertainty in the model’s predictions.

The Motivation
As black-box models, LLMs are also criticized for their lack of interpretability. The specific patterns and functions LLMs used to arrive at predictions or decisions are not directly accessible or explainable to humans. Even though some LLMs are equipped to explain their predictions by applying chain-of-thought, their reasoning explanations also suffer from the hallucination issue.

To address the above issues, a potential solution is to incorporate knowledge graphs (KGs) into LLMs.

Note. From Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2023, June 14). Unifying large language models and knowledge graphs: A roadmap. arXiv.org. https://arxiv.org/abs/2306.08302

What are Knowledge Graphs?

A knowledge graph consists of a collection of facts, represented as triples. Each triple contains three components: a “head entity,” a “relation,” and a “tail entity.”

The typical approach we see nowadays, where we simply chunk and embed documents, might fail when looking for information that spans multiple documents. This problem is also known as multi-hop question answering. However, the multi-hop QA problem can be solved using knowledge graphs. One way to look at a knowledge graph is as condensed information storage. Using knowledge graphs, you can represent highly-connected information that spans multiple documents as relationships between various entities.

Let’s write some code.

In this example, we assume the existence of a Knowledge Graph that contains information about movies and actors. The goal is to retrieve relevant context information from the Knowledge Graph based on the user’s question.

from neo4j import GraphDatabase

def retrieve_context(question):
    # Connect to the Knowledge Graph
    driver = GraphDatabase.driver(
        "neo4j://localhost:7687",
        auth=("neo4j", "password")
    )

    # Define the query to retrieve context
    query = """
    MATCH (m:Movie)-[:ACTED_IN|DIRECTED]-(p:Person)
    WHERE m.title = $movie_title
    RETURN p.name AS actor_name, m.released AS movie_year
    """

    with driver.session() as session:
        result = session.run(query, movie_title=question)

        # Process the result and extract the context information
        context = []
        for record in result:
            actor_name = record["actor_name"]
            movie_year = record["movie_year"]
            context.append(f"{actor_name} acted in a movie released in {movie_year}")

    # Return the retrieved context
    return context

Credits to: https://github.com/tomasonjo/

The retrieve_context function defines a Cypher query to match the desired pattern in the Knowledge Graph. Depending on your domain, you might also use custom queries that retrieve information more than one hop away or sometimes want to aggregate some results.

You can check out Neo4j’s GraphAcademy to learn more about Cypher query language.

Nevertheless, KGs are difficult to construct, and current approaches in KGs are inadequate in handling the incomplete and dynamically changing nature of real-world KGs. Also, Conventional LLMs trained on plain text data are not designed to understand structured data like knowledge graphs. Thus, LLMs might not fully grasp or understand the information conveyed by the KG structure. Therefore, it is necessary to develop LLMs that can directly understand the KG structure and reason over it.

And there you have it, a glimpse into the fascinating world of integrating Large Language Models with Knowledge Graphs.

References

Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2023, June 14). Unifying large language models and knowledge graphs: A roadmap. arXiv.org. https://arxiv.org/abs/2306.08302
CS224N: Natural Language Processing with deep learning. Stanford CS 224N | Natural Language Processing with Deep Learning. (n.d.). https://web.stanford.edu/class/cs224n/
Please upvote if you like my work and comment your feedback!

Connect with me:
LinkedIn: https://www.linkedin.com/in/shivam017arora
Twitter: https://twitter.com/Shivam017arora