Shared Misunderstanding: Where representations in Large Language Models go wrong.
Welcome to a set of blogs looking at the intersection of language, knowledge representation and AI. In this post, we’re going to take a deep dive into some of the concepts of Prototype Theory, and ask if the advances in AI and NLP can provide any new tools to investigate these cognitive phenomena.
Imagine a “piece of furniture”. What do you see? A “chair”, a “table”, a collection of “different pieces of furniture”? Or a “half-table, half chair, all impractical monstrosity”, which kind of resembles a chair if you squint, a table if you cross your eyes?
Most likely you’ll have imagined some kind of furniture. For me, it’s a chair. In my mind, a chair is the prototypical piece of furniture. Your prototype could be different.
Now, let’s imagine that chair. It’s easy if you try. It’s instantly recognizable. A table too; and a stand-lamp. They’re all easy and natural to imagine.
So, what is the difference here? What sets apart “a piece of furniture” from a “chair” and a “table”.
Chairs and tables seem more fundamental, more “real” than furniture. Cognitive Scientist and psychologist Eleanor Rosche calls these more fundamental objects “Basic level categories”, a key concept in Prototype Theory, which we will dive further into in another post.
Basic Level Categories are real, they are defined mostly in terms of sensory-motor affordances, or how we interact with physical objects. Everything else, such as abstractions and hierarchies, are a feat of the imagination. These abstractions are not inherent in any object but are created for our convenience. There is no real thing “furniture”, there are only the basic level categories of chairs and tables, etc.
This is why it is easy to picture a chair, but when picturing a piece of furniture, we default to picturing different types or instances of furniture.
The way we categorize the world around us - and the way we build hierarchies of the things in the world - is core to how we interact with, internally represent and externally communicate about the world. The hierarchies we come up with make communication and reasoning about the world much more efficient.
The Role of Hierarchies in Understanding
These shared hierarchies of the world allow us to communicate with incredible efficiency, allowing the shared hierarchy to do the heavy lifting in a conversation.
Do you have any furniture at home?
Answering this question depends on the shared understanding of what constitutes furniture. In this case, we’re talking about a hierarchy which maps the abstract concept furniture to all of the types of furniture below it. Without this shared hierarchy, in a linguistic flat-land, we would have to enumerate all the things we believe to be furniture, to get the exact answer we expect. If you’ve ever been to the Bauhaus Furniture museum, you would know how tedious this will become.
As you can imagine communicating without hierarchies is very precise, but it is incredibly inefficient. Hierarchies allow us to template language to gain both precision, and efficiency. Even if our shared hierarchies are not perfectly aligned, the benefit of efficiency is worth the cost of absolute precision. I, for instance, would disagree with Wikipedia that a Hi-Fi counts as furniture.
The Missing Text Problem
The Missing Text Problem is the idea that in order to understand any piece of text, there is unwritten information that the reader and writer share, which cannot be inferred from the text alone, but is crucial in the full understanding of that text. Shared hierarchies provide insight into a partial solution to this problem.
Experiment: Does AI approximate Human Level Hierarchical Understanding
When reading about Basic Level Categories in “Women, Fire, and Dangerous Things: What Categories Reveal About the Mind”, I immediately thought about how the methods we use to demonstrate the intuition of the concept rely heavily on both imagination, and communication.
With the advent of AI, we can start to ask, do machines follow the same abstractive patterns that we believe humans follow? After all, the explosion of modern AI is built upon the promise of better representations, using neural networks to build powerful “abstractions” for improved language understanding.
So, the natural question to ask is what does an AI imagine when we ask it to imagine chairs, tables, and furniture? Does it represent furniture as a selection of the instances that make up furniture, or does it not represent furniture as an abstract concept at all?
Luckily, with Natural Language-driven image creation AIs such as Dall-E we can begin to put this to the test.
Let us start with the basics, to reassure ourselves that AI representations can indeed create credible imaginings from language alone.
Here we see that when asked to imagine a chair, we get nine very credible chairs. (Not one chair that I would hesitate to sit on, although top-right is a little suspect). We can go further, as ask to make tables as well.
We can see that the AI has a good grasp of tables, with some small interesting elaborations, such as the centre table, which creatively merges with the curtains, and a possible chair.
One could anticipate that when DaLL-E is asked to represent the general concept of furniture. If it follows the principle of abstraction and Basic-level Categories, we should see a selection of different kinds of furniture. But what we see is much more “Dali-esque”.
What we see is that the internal representation of furniture is being interpreted in a way that results in weird and wonderful hybrids. Some, such as the bottom right, are fair examples of individual pieces of furniture, recognisable to us. However, in the bottom center, we see a frankensteinian chair-table combination, which breaks with the human approach to abstract visualisation; instead it attempts to combine all the definitive features of things considered furniture into a nightmarishly impractical joke.
What does this tell us about the internal representations used in these models?
Here, the representation of furniture has aspects of all the things it has seen that are furniture, creating an “average” set of furniture features. Good features for sure; intelligent and recognisable features, such as handles, surfaces, legs and draws. The amalgamation of these features maps quite nicely onto the metaphor of family resemblances spoken about in Prototype Theory, however that is where the similarities end. Internally, the representation of the word “furniture” is an average of all the furniture, not a collection of prototypical furniture. The machinery used to build these representations has failed to internally create any meaningful hierarchies.
This is innate in how embeddings work. Everything is embedded into the same space. There are no real abstractions being made. This representation is useful, but it is not understanding. It can help us build great classifiers, but it doesn’t really help us understand how humans think.
To me, this motivates the question of what we need to do next, to improve knowledge representation in AI models, hierarchies must play a role.
A Shared Misunderstanding Between AI
We mentioned that to understand text well, there is a wealth of missing information that we as humans have internally and generally share. We rely on shared understanding for efficient communication, and at least one part of that understanding is ontological hierarchies. From this understanding, we can reasonably ask: Do these AI models have any shared understanding? Can they communicate ideas and be understood between themselves? Here we complete that circle.
In case furniture isn’t a riveting example to you, we can see how the same system butchers abstractions, this time with animals.
These grotesque approximations of animals again show how the models used by DaLL-E do not handle abstractions well. In fact, not a single one of these are recognisable as an animal. When I asked a biologist to name these creations I got: Scampir (my personal favourite), Dongang, Cog, a Tigeroo, and Shion, among others.
But, if you ask an AI trained in zero-shot classification to tell you what it sees, it will confidently assert that this is an animal; just not a specific one.
On a technical note, this is fascinating. The AI making this decision (FLAVA) is a separate AI than the one that built the image (Dall-E), but they steadfastly agree, that this is absolutely an animal. Both AIs trained with different data, extracted universal properties, and have come to the same conclusions. In a sense there is a shared understanding that has been abstracted from the data: the core features of an animal.
One AI imagined a never before seen animal and the other AI also understands it to be an animal.
On one hand, this is a triumph of shared representation through different exposure. But on the other hand it is terribly, terribly wrong. What we see is an abomination, not an animal.
While a potentially silly example, it exposes an issue this these representations, and brings the story back to linguistics, the power of abstraction and base level categories.
We’re asking about ontological types and we’re being answered with statistical correlations. This basic miscommunication is a red flag when it comes to trust in AI models.
When we ask, “Is this an animal?”, we’re not asking, “Does it share enough features to resemble an animal?” We’re asking, “Does this belong to the super-class animal?”
Without learning to represent the hidden hierarchies in language, these systems cannot understand or answer the logical question we are actually asking.
