AI’s Woolf at the door — LLMs and Knowledge Graphs

I was at the Knowledge Graph Conference this week, and as usual, it was a fun and insightful time. I have enough topics for at least four blog entries, so I’m going to start working through them.

This first one is about a topic that seemed to be on just about everyone’s mind, whether the recent explosion of interest in AI and LLMs was going to be finally the turning point that would put Knowledge Graphs on the map, because, as some have said, and many at the conference were discussing, Knowledge Graphs and LLMs are “a match made in Heaven”.

Now, I’m guessing that this is happening in every conference of any sort everywhere; that our favorite topic is the perfect partner for LLM. Electronic Health Records and LLM? Molecular biology and LLM? Social work and LLM? Project Management and LLM? Many of these I’ve already seen, and I bet there are dozens more; my favorite topic of interest is the perfect partner for LLM. And as such, we at the Knowledge Graph Conference are just joining the bandwagon, and saying the Knowledge Graphs are the perfect partner for LLMs. So let’s see; do we really have any better argument than all the other suitors at the dance who see themselves as a perfect match for LLMs?

There are a number of ways to expect this marriage to work. The one that makes the most sense to me is some form of cooperation between the KG and the LLM. Keynote speaker Deb McGuinness seems to agree, since she collaborated with ChatGPT to create the slides for her talk, where the one thing she asked us to takeaway, if nothing else, was that we should expect to see successful applications of LLMs be collaborative applications, where the LLM works together with a human in some way. This notion of having a human who can take responsibility resonates with some of the learnings from the age of Expert Systems, where we realized that an expert system needs to work with a human who can take responsibility for its actions.

It seems to me that we have already seen a number of examples of the most productive way for humans to collaborate with LLMs; I’ve done it a lot in this blog, where I have asked an LLM to help me write a program that will present the results of the queries I am posing to the knowledge graph; probably the most elaborate example of that was the heat map results of the Syngenta Good Growth Plan; the heatmap is shown below.

How did I create that heatmap? I talked to ChatGPT and told it I wanted a heatmap of the world, I told it I wanted earth-tone colors, darker for higher numbers, and told it (in English) what I wanted it to show. It gave me this diagram. I decided on the question I was interested in, how I wanted to show it, and ChatGPT did all the leg work to display it.

What I did isn’t very special; Sean Martin has taken this one step further, and made a feature in Anzo (the “Data Guru”) that lets a user converse with the bot, and create data visualizations, adapting the query and the visualization side-by-side. His demo isn’t slick or flashy, but boy, is it impressive. You talk to the bot about your data, and it is able to display it using all sorts of tools.

But impressive as that demo is, and to take nothing away from Sean, my own data.world colleague Bryon Jacob showed me the same demo a few weeks ago; Mike Grove at Stardog showed a demo that did much the same thing. The key part of this is “obvious” in the technical sense; several qualified professionals (myself, Bryon, Sean, Mike and I’m sure a bunch more) have all come to the same conclusion and have all reduced this to practice in various ways. It seems pretty clear to me that the puzzle of how to make your data understandable in visual and compelling ways has just been solved in a much deeper way than ever before.

In his keynote, Denny Vrandečić made a point that I think is very important; he asked a simple question (“who is the creator of ‘The School of Athens’?”) to three information agents; ChatGPT, Google, and Wikidata. All three got the same answer; they interpreted this as the renaissance fresco, not as some Athenian philosophical school, and correctly identified Raphael as the painter. But he measured the performance of al three; ChatGPT took longer than Google, which took considerably longer than Wikidata. And, as we have all experienced before, the LLM might not even get consistent answers (and neither is Google guaranteed to do so, but for something like this, it will almost certainly get the right answer). There’s something very important to notice here; ChatGPT and Google are “figuring out” the answer, where as Wikidata is just repeating the answer that it was “told”. I’ve written at some length about “figuring out vs. telling”, and this is a great example; but Denny points out that figuring out has two disadvantages in this situation; not only is it less reliable, but it also costs a lot, lot more.

So, if you bring this all together, it seems that we have our case; the knowledge graph is the place where you can “tell” (and then eventually “ask”) the things you know, and the LLM is the place where you can “figure out” the things you want to know. There’s our division of labor, and the basis of our marriage! But the title of this blog (which I collaborated on with ChatGPT) refers to a ‘successful’ marriage that, behind the curtains, is anything but.* I am still skeptical, for a few reasons. First off, I don’t want to be like all the other suitors, who think they are the most suitable; I want to make sure I have a strong case for the marriage with ChatGPT. Second, I think it is a scientist’s job to be skeptical. And while I might not really be able to call myself a scientist, I do want to remember how to act like one from time to time.

Is it really a match made in heaven?

So, what is my skepticism based on? As impressed as I am with Sean’s Guru, I can’t help but look at some of my own experiments, with the Syngenta Good Growth Plan data. That data is basically a set of about half a dozen spreadsheets. Yes, I loaded them into data.world, so they are in a graph database. But did I really use the fact that they are in a graph when I queried them? For the summarization task, I actually un-graphed the tables; the prompt to ChatGPT was a list of names of columns, the name of the table, and the text that describes the whole dataset. I completely removed the graph from the equation. In the blog about LLM Hallucinations, I show a transcript of a discussion with ChatGPT where I ask it about the data, then I make a follow-up question about the results of the previous question, and follow-on with a very complicated question. ChatGPT gives smooth answers right away, much as they did in Sean’s video above. And yes, I asked it to write SPARQL, but it wrote those queries over a single spreadsheet; there was no linked data being invoked, no use of Things vs Strings, etc. I could have had it write a SQL query just as easily.

So this leaves me wondering, is it KG that should marry LLMs? Or just simply databases? I want to consider some of the things that a KG can do that a database alone cannot, and see whether any of these can help the LLM be more effective in its work.

1. In a Knowledge Graph, you can write down (“telling” in the “Figuring out vs. Telling” sense) the fact that one entity is the same as another. This is usually important when collecting linked data about something. The customer who purchased this product is the same as the person who filed this complaint, etc.
2. In a Knowledge Graph, you can represent metadata and data in the same format, in the same source. This means you can query the metadata and use that result to query the data.
3. In a Knowledge Graph, you can augment a dataset easily, by superimposing a new graph on top of an old one. This works best when you have solved the identity problem (#1 above) in some form, so you know what has to superimpose on what.

In some sense, you can do all of these things with a database, but to do so, you will basically have to reinvent something like RDF (global identifiers linking to other global identifiers with identifiable links); I think these three features really are essential to differentiating a knowledge graph from a database. There might be more differentiators; they might be useful for LLMs, too.

This blog is already pretty long, so I’m not going to try to answer the question, for each of those bullets, “Is there something special about this capability that will make KGs more useful as LLM partners than simple databases?” Also, I don’t know the answers for most of these; I think these are points for further experimentation (for those days when I pretend to be a scientist).

For #1, it seems that the ability to record identities would really shine in a situation where we have identity services already working. There are a lot of approaches to identification; registries (like the GLEIF, the SEC, the SIB, Chemical Abstracts, etc.), machine learning, heuristics, etc. Can an LLM perform better when it doesn’t have to figure this out for itself? This might be a difficult set of experiments to design; if you have any ideas, I’d love to hear them.

For #2, this is a bit easier; those of us who use Knowledge Graphs tend to take it for granted that you can add a lot of metadata to a column header of a spreadsheet (in a real spreadsheet, this is wicked difficult). What sort of metadata? Comments written in natural language (data dictionary style), linkages off to controlled vocabularies (general ones like schema.org or specific ones like FIBO), and even groupings (these five fields together form an address). Can we get more reliable query generation if we include this sort of metadata?

For #3, this could be quite the experiment for a linked data catalog. Suppose you try to have ChatGPT write a query across multiple data sets; how can it know how to join them, without knowing an answer to the identity question? Could “telling” it those “joins” in a knowledge graph help it write more insightful queries?

I think I have some experiments to do for future blogs. Stay tuned.

*The title refers to Edward Albee’s play, “Who’s Afraid of Virginia Woolf”, where we see a married couple bring other people into their internal bitter strife. As an aside, I love discussing literature with ChatGPT. I’ll write some of that up in a future blog.