This is a short, fun, and not super-rigorous article where we delve into the consideration of whether LLMs “think” or “reason.” I am going to take some ideas from formal languages, and formal systems, and consider them in the context of AL, and in particular LLMs (Large Language Models)

We should though point out that “reason” is a term that’s used very loosely, often we think of this as considering data and logical statements and synthesizing them into a conclusion. This is often done using rules of logic etc. (but maybe this actually is not the case, more on that later)

What is a formal language?

A finite or infinite set of strings, considered in isolation from any possible meaning the strings or the symbols in them may have. If A is any set, an A-language (or language over A) is any set of A-words (See word). A is referred to as the alphabet of such a language.

So if we take this and consider the operation of adding and subtracting words from the sentences in the language we can get what is referred to as a formal system.

A formal system is an abstract structure and formalization of an axiomatic system used for deducing, using rules of inference, theoremsfrom axioms by a set of inference rules.

In 1921, David Hilbert proposed to use formal systems as the foundation of knowledge in mathematics.

Assuming that this formal language of human words (tokens) satisfies the criteria for the Gödel incompleteness theorems (they don’t quite) on formal systems, then this system wouldn’t have the capability to consistently derive all possible results from it’s axioms, but that wouldn’t mean that it cannot reason – it just means that it cannot reason everything. And that’s something that has plagued scientists well before chatGPT started trying to get a fox, a hen, and a sack of corn across a river.

Maybe there’s no inherent barrier to LLMs reasoning?

PS: The Gödel incompleteness theorems loosely state that in a rich enough logical system there will be true statements which are not provable from just the axioms (something like the continuum hypothesis)

Why does any of this fancy talk matter?

The data fed into LLMs are tokens without inherent meaning, and then based on positioning within text, a complex network assigns weights as to the relationships observed, ie human beings produce text and the network is trained to replicate the patterns of the text.

Ok, so what?

Humans use different forms of logic, but they more or less boil down to us writing, visualising, speaking, etc logical statements.

But where does that come from?

All of what mankind produces comes from the mind, from the human neocortex – a more advanced neural network – in fact what we perceive as thinking are parts of the brain’s network firing up. So what’s to say that what appears like conscious logic to us, isn’t in fact pattern matching within a superior neural network? (well superior to what we can produce now)

So we might ask why would the formal system of human language trained on human usage (where we make meta analyses of language all the time) not be able to reason?

An argument made against LLMs is that they make errors in logic by following patterns, but isn’t that what we teach students to look for in solving problems, look for the pattern, and modify? The errors made in modifying the pattern don’t prove that LLMs aren’t reasoning, they just lend evidence to current LLMs reasoning poorly. That doesn’t mean they can’t.

Indeed we reasoning human beings can make errors, if you’ve graded any first year college courses in logic or mathematics you’ll be aware of that.

The problem with this, and maybe the question itself?

Is that without a precise definition of “reasoning” (and indeed it would likely need to be one that almost all human beings fall into) we can’t falsify the statement that LLMs can reason on the one hand, and on the other we can’t falsify the statement that they don’t.

I think we often tend to fall on the side that LLMs cannot reason because they don’t look, feel, sound (well that last one is changing) like humans that they cannot reason – but is that really a scientific point of view?

Maybe until somebody much smarter comes along and proves one way or the other, perhaps it’s worth considering that LLMs can reason, and with sufficient power could reason at or above human ability, but of course what is missing what makes it seem not to reason, is will?