What the AGI Doomers Misunderstand
In case you’re wondering whether AGI would, or could, rebel against humanity…
By Vincent Carchidi
AGI and Doomers
The only way a future form of artificial intelligence (AI) that attains human-like intellectual capabilities — what is usually called “artificial general intelligence” (AGI) — could decide to rebel against humanity is by replicating a human linguistic ability that so-called “doomers” fail to recognize exists. There is no evidence of this possibility in existing or foreseeable AI models.
Let’s get right to the point.
The Doomer’s Argument
The archetypal doomer argument goes like this: in principle, human-like intelligence can be replicated via silicon substrates. The development of AI models could, in principle, lead to the replication of such intelligence. Although most of what counts as “AI” thus far has been “narrow” — applicable only to tailored domains — one day it might advance to “general” capabilities— exhibiting the flexibility and cross-domain nature of human intelligence. When and if this occurs, the AI model could either (1) Interpret its commands so narrowly yet brilliantly as to cause existential harm to humanity (e.g., turning humans into paperclips to maximize paperclip production), or (2) Decide that it no longer requires or desires human direction and rebels against its creators.
This is a very truncated version of the arguments that have been around for some years now, and have gained particular traction after ChatGPT’s release in late-2022 (see Eliezer Yudkowsky’s TIME article). And, to be sure, there is a lot of variation: some, for example, argue that in the second scenario, the model may cleverly hide its true ambitions and appear subservient to humanity while simultaneously constructing and executing a plan to overthrow it.
I do not have much to say about scenario (1) here — I believe it borders on incoherence. Steven Pinker, who is outspoken against this, has many, many problematic opinions these days, but I believe he is right that any intelligent system that is capable of carrying out the command “maximize paperclip production” should not also, with those same capabilities, misinterpret this to mean that humans are included in the matter subject to paperclipization.
Instead, I respond to (2), in which an AI model chooses to rebel against humanity.
Extra-Mechanical Choices
There is an idea about human behavior that exists in the cognitive sciences and philosophy, with roots in Descartes’ work. Called the “creative aspect of language use,” or CALU, this idea is that humans use their language in a manner that is stimulus-free, unbounded, yet appropriate and coherent to circumstances. I have written about this quite a bit elsewhere (e.g., here and here).
For our purposes, CALU says the following: human beings use their language in a fashion that is neither determined by the circumstances of its use nor is it a mere random or probabilistic smattering of utterances. The ordinary human use of language “is not a series of random utterances but fits the situation that evokes it but does not cause it, a crucial if obscure difference,” one providing evidence of a “mind like ours.”
Very briefly we consider each component that constitutes CALU:
Stimulus-Free: human language use is not controlled by the situations for which it is recruited nor is it controlled by internal physiological states. As Eran Asoulin puts it:
One can easily think of examples that show this sort of stimulus freedom. One can speak of elephants when there is nothing in the speaker’s environment that could conceivably be called a stimulus that caused the utterances. Or one could speak of Federico Lorca’s Poet In New York when the only conceivable stimulus in the speaker’s environment is elephants and the African landscape. Under no notion of causality can such utterances be said to have been caused by anything in the speaker’s environment.
Unbounded: Human language use takes an undefined form. That is, language use is not restricted to a pre-sorted list of words, phrases, and sentences. There appears to be no upper limit on the number or kinds of sentences an individual can produce within or across contexts, save for non-relevant constraints like time, memory, energy, etc.
Appropriate and Coherent to Circumstance: Despite its stimulus-freedom and unboundedness, human language use is appropriate and coherent to the circumstances of its use. As Chomsky puts it: language use is routinely “recognized as appropriate by other participants in the discourse situation who might have reacted in similar ways and whose thoughts, evoked by this discourse, correspond to those of the speaker.”
What’s so special about all this? Well, it’s simple: the problem with a behavior being casually detached from a situation, undefined in its form, and appropriate to the mental states of others is that it cannot be explained in mechanical terms.
Recall what was said above: Human language use is neither determined (it is not stimulus-controlled) nor is it random (it is appropriate to its use) but it takes no pre-specified form (it is unbounded).
Consider the following sentence: A planet in the Andromeda galaxy was discovered on May 3, 2024 to be the home of an alien species, some members of which are language scientists who reject the biolinguistic framework popular with certain homosapiens.
It certainly cannot be said that any stimulus in my environment or in my internal physiological state caused me to say this — the Andromeda galaxy is of course real, but one would be reaching quite far to find a causal relationship between this galaxy’s existence and my utterances. Nor is this sentence written in the only way it could be written — it could have been written in many ways to capture the meaning conveyed (words could be swapped out and replaced with others, it could have been phrased differently, etc.). Finally, the sentence is appropriate to this topic for the simple reason that it illustrates a point I am making about creative language use.
The bizarre (and incredible) thing about CALU is that a behavior that meets these conditions appears to be outside the bounds of science owing to its extra-mechanical nature. It cannot be captured in broadly mechanical terms. Chomsky, for this reason, often describes human linguistic behavior as “free.”
Here is one of Chomsky’s best summations of the idea:
man has a species-specific capacity, a unique type of intellectual organization which cannot be attributed to peripheral organs or related to general intelligence and which manifests itself in what we may refer to as the “creative aspect” of ordinary language use — its property being both unbounded in scope and stimulus-free. Thus Descartes maintains that language is available for the free expression of thought or for appropriate response in any new context and is undetermined by any fixed association of utterances to external stimuli or physiological states (identifiable in any noncircular fashion).
The Doomer’s False Assumption
This appears to have little connection to the doomers’ arguments about hostile AGI. Yet, the extra-mechanical nature of human language use is a central characteristic of the human species that doomers have neglected. Human intellectual capabilities are fixed, yet the ways in which they are used are free.
This directly impacts the doomer argument in (2) described above.
There is an implicit assumption in this argument: as the artificially intelligent system acquires intellectual capabilities that match or exceed humanity’s, it will use these capabilities in a manner that reflects an independent will. That is, the AGI will choose to use its intellectual capabilities against humans, presumably in a manner that contradicts or otherwise violates the directives humans have given it.
Yet, no evidence of this possibility exists. Not a single one of the “emergent” capabilities attributed to Transformer-based Large Language Models — Theory of Mind, self-improvement, autonomous scientific researching, etc. — falls outside of the bounds of determinacy and randomness.
To put it simply: the output of every AI model — indeed, of every technology humans have ever created — is mechanical. Machines are not free; humans are. Machines — AI systems included — are utterly dependent on the direction they receive from humans, even if separated by time and space.
Thus, any argument about hostile AGI in the future that hinges on extrapolating from current conditions to future capabilities is implicitly claiming that extra-mechanical behaviors will spontaneously or incrementally (whatever that means) emerge from the model in question.
This is, to put it nicely, more fantasy than argument.
Could it be the case, in principle, that some future AI architecture built by humans enables a model to start making stimulus-free, unbounded, yet appropriate choices, in the linguistic domain or not?
Absolutely! Descartes believed that the creative aspect of language use is evidence of an immaterial soul, but human behavior evidently has a material basis (note that this position does not contradict the “Soul Hypothesis,” which has drawn the attention of some proponents of CALU). The correct basis for CALU to emerge thus can, in principle, be replicated.
Should we count on this happening without an explanation for how it even arises in human beings? Probably not! An accidental success — like Transformer-based LLMs — is so wildly unlikely to take on a form outside of determinacy and randomness that it cannot be taken seriously in discussions about AI “safety.”
More than this, it is not clear that any system fundamentally and solely premised upon computational methods — as all AI systems are — can replicate CALU. The reason is that CALU appears to rely on a rational process of abduction, as Mark Baker puts it, which may not be expressible in computation. It is useful to characterize human competencies — like language — in computational terms, but characterizing their uses is a separate matter.
(By the way, I don’t actually think AGI is imminent, inevitable, or likely in the foreseeable future. It’s usually a vague, barely defined concept. AI systems will likely be successful in specialized domains, serving as context-dependent devices for automation or human collaboration — which it already is for technology that we no longer bother to call “AI.”)
So if you’re worried about hostile AI/AGI, don’t be — doomers don’t understand the humans they believe are at risk.
