ChatGPT is a Chinese Room!

8 min readDec 18, 2022

That’s a problem and you should know why.

What Stable Diffusion produces when asked about Chinese Rooms

The last week or so (early December 2022) has seen lots of excitement about the public interface to a Chatbot called ChatGPT that OpenAI has created. Apparently, This is a conversational interface hooked up to an improved version of the GPT-3 language model. A fine tuning and reinforcement learning algorithm are used to improve conversational performance vs. one shot text generation. You can read more about how it works on the OpenAI blog post.

In this blog I contend that ChatGPT is an example of a Chinese Room (this is a machine postulated by John Searle that answers questions without understanding them). I explain why this is a problem both because it sharply limits how useful ChatGPT or similar systems are, and also because of the danger it creates for people (and organizations) that try to exploit this technology for inappropriate applications.

TLDR; it doesn’t understand what it’s saying, and you can’t trust it in an application.

The excitement in social media, and the mainstream media has been intense because ChatGPT can be used by anyone who creates an account. This is in contrast to previous models created by OpenAI that haven’t been released into the wild. People are able to play with it and ask it questions and judge it’s performance for themselves. Many people are dazzled because ChatGPT is very human like in it’s behavior and very good at retrieving information on some interesting topics .

I found it to be particularly strong on programming. For example, I asked it a question relevant to a problem my team was having a few weeks ago where we ran some legacy code against a bigger more complex data source and hit a scaling issue:

Some great programming advice from ChatGPT

This is the sort of advice you can get from sites like Stack Overflow, but often finding these approaches requires quite a lot of search skill and patience. Typically there are lots of references to questions that others have asked but are badly answered and these hide the gems that have exactly the right information to solve the problem at hand. ChatGPT (at least in this case) seems to cut through that process.

So what’s the issue with other applications? Reduced to its bare bones ChatGPT is several very large sets of floating point numbers. These sets are called deep networks and sometimes deep neural networks which makes them seem like a biological process — when in fact they are just millions and millions of numbers fed into relatively simple functions that create the outputs that we see. These numbers are combined to create complex descriptions of the patterns in language and these patterns are combined with the inputs that a user provides to output the text that ChatGPT outputs. While a technical understanding of ChatGPT can be got from reading the OpenAI blog, it’s useful to think about what it’s doing in an analogical way as well.

Imagine it like this: the patterns in the deep networks of ChatGPT are like a vast room full of billions of little index cards, each of which has a word or two, or three written on it. Messages drop in from the ceiling of the room and are collected by a robot and taken to a huge book which contains a lookup table. The robot starts by looking up the message. It finds an index entry which is either the message or something very close to it and the index points to the location of a card. So the robot wizzes off, and looks at the card. The words on the card are added to the message and this is now used as an index in the book. This process repeats until the instruction for the index says “stop” and then the robot removes the original message from the words that it’s put together and throws the new message that it’s created out of the door of the room.

When viewed with this analogy (which is as close to what is going on as saying “it uses machine learning”) ChatGPT seems to be very close to the classic formulation of a Chinese Room — introduced by John Searle.

The robot, and the room (the system — ChatGPT in this case) don’t have a clue what the message that they received means. The same is true for the message that the robot created and threw out of the room — nothing in the system understands that message in any way.

ChatGPT allows us to show that it really is possible to build a working Chinese Room which demonstrably has no understanding of what it’s talking about. Previously people argued that it wasn’t possible to have a Chinese Room without it having actual understanding. To prove it here’s an example that demonstrates that the ChatGPT version of a Chinese Room really doesn’t understand anything.

A dialog where ChatGPT shows it doesn’t understand when it doesn’t understand

I gave it the answer, it agrees with me, but then it can’t remember what it was told.

So in this dialog ChatGPT is given a simple problem to solve, it outputs an incorrect solution (1) , when challenged it outputs some complete nonsense and doubles down on it’s error (2). After that I give it the right solution which it goes along with… but then when I ask it again it goes back to the beginning.

The point is that the little robot, the magic book and the room filled with little index cards have no idea what they are given as input, or what they are creating as output. This is because all we are seeing is mathematical approximations of conversations that it’s been train on that appear to be reasoning processes, but ChatGPT *never* reasons about anything, because it can’t.

Can’t figure out the cost of a trip either — why does it stick with $600?

This is not a problem so long as everyone shares the same beliefs about what ChatGPT is and how it works.

The problem is that many people, confronted by ChatGPT don’t think of it as a Chinese Room, only geeky people like you and I have even heard of Chinese Rooms. Instead a lot of people, including decision makers with lots of money are going to look at systems underpinned by ChatGPT or similar engines and are going to see them as HAL, or at least as a “minded” machine. We’re evolved to do this, a rock or a tree is not a dangerous thing. Something with a mind is potentially very dangerous and so we’re predisposed to identify these entities and treat them as things that think.

So, most people when confronted with something like ChatGPT are going to instinctively think that it’s reasoning about things that it understands, and that it has purpose and agency to choose how it reasons and what features of the input it reasons about… just like a wild dog on the savanna would. Their relationship with ChatGPT will be formed on this basis, just as it would with a dog, but it’s a completely wrong basis for thinking about these things and dealing with them. They are not minded like dogs, or any other animal or person. They are not minded at all.

Other very clever people have considered the question of how we talk about ChatGPT and similar systems [1]. From my point of view the danger is that if you create a demo of an application with an engine powered by a ChatGPT class you can probably tweak it to run lots and lots of times before you hit a show stopping response. It’s going to do some impressive seeming things. You can use these demos and a convincing narrative about your extensive testing to convince people to invest or support your application or whatever.

I expect lots and lots of people are feverishly developing applications providing customer service, mental health counseling or advice for cancer patients. I think that the dangers of advising people to refuse to pay, take more drugs or follow a homeopathic approach to treatment are pretty obvious. That’s the kind of thing I can imagine a mindless machine, or a Chinese Room doing — and not even being able to realize it did it.

So, the question is what applications can a ChatGPT like system actually be used for? Here are some ideas:

Code coaches/assistant. This one is not without dangers, it’s easy to imaging people getting duff advice from ChatGPT, but then again you can get duff advice from StackOverflow and waste a-lot of time with that. I think though that with enough caveat-emptor programmers will (and do) find this sort of assistant very useful. The fact is that for programming the code is the law; if it works, it works. You can’t fool the compiler or the virtual machine so if ChatGPT screws up you will find out immediately and the damage will be on you.
Foiling scam callers. ChatGPT can be harnessed to have convincing knowledge free conversations with scammers on mail, chat or the phone wasting their time and damaging their economic model. If the bot spouts nonsense at them who cares?
NPC dialog in video games (with extensive output filters to prevent “Oscar the Blacksmith” spouting neo-nazi propaganda).

But, really that’s about it. I don’t think this sort of engine can be trusted to power an FAQ type chatbot, or to summarize text, or to produce index entries (I tried this one for my book, but it hallucinated a bunch of topics that weren’t in the text I gave it).