Do Generative AI chatbots have intention? Considering a question raised over 40 years ago in Against Theory
In this piece I make the case for the following: that intention can be (and long has been) mechanically constructed in computationally produced language. This is precisely what we observe in the outputs that we receive from large language model (LLM)-based chatbots. Despite the sophistication of their responses to human prompts, such chatbots are not conscious or sentient (at least not yet) and do not operate with “intention” in the way that we might think about that concept for human-produced language. Nonetheless, I show that intention — functionally speaking — is mechanically constructed in the computation systems that orchestrate LLM-based chatbot outputs. On its face, this proposition may seem straightforward if not obvious. However, for over 40 years a debate has raged in literary criticism circles regarding the role of intention in interpreting texts or other speech acts. That debate has recently resurfaced given the emergence of Generative AI. Perhaps somewhat stuck in the polarities of the old debate, commentators seem to be struggling with how to explain what we experience when we interact with LLM-based chatbots. To illustrate my explanation for why LLM-based chatbots exhibit human-like intention in the outputs they generate, I use the example of a player piano.
Background
In 1982 Steven Knapp and Welter Benn Michaels, two professors of English Literature, created quite a stir with the publication of their essay, Against Theory (“AT”). Over the latter part of the 20th Century many theorists had proposed a variety of methods for conducting literary interpretation, typically proposing alternative means for determining authorial intent, and sometimes questioning the role of the author in determining the meaning of a text. In response, the authors presented a simple proposition: there is no need to separately consider the matter of authorial intent when considering a text; the language presented by the text itself is sufficient for understanding the meaning sought to be conveyed by its author. This proposition provoked significant controversy. Not only had alternative views on literary interpretation proliferated in the prior two decades, but Knapp and Michaels used their simple proposition to deny that the enterprise of literary theory was at all worthwhile. The essay largely presented a series of refutations to leading theories (and theorists) of literary interpretation and certain theories from philosophy. The authors also used a thought experiment to help prove their point. They presented the concept of a wave poem — a set of inscriptions in the sand that appear at a beach after a wave washes up and recedes. The inscriptions happen to resemble a short William Wordsworth poem. In Knapp and Michaels’ view, if the inscriptions were created purely by accident, then this would be an example of intentionless language and it would make no sense to ascribe any meaning to the inscriptions. This is the corollary to their basic proposition: just as one need look no further than language produced by an author to understand the author’s intent, unless there is an actual author behind language that one encounters, then one cannot impute intent or meaning to such language. Over the years that followed, AT inspired a vivid set of exchanges with other thinkers in the field of literary theory, leading to many essays and a few books, including rejoinders by Knapp and Michaels to the many critics of their position. I recall reading these as an undergraduate student at Berkeley.
AT was recently the subject of an online forum convened by Critical Inquiry, the original publisher of the AT essay. The occasion for the forum was a passage from AT raising the question of whether computers can produce language reflecting intention:
“But there are cases where the question of intentional agency might be an important and difficult one. Can computers speak? Arguments over this question reproduce exactly the terms of our example. Since computers are machines, the issue of whether they can speak seems to hinge on the possibility of intentionless language. But our example shows that there is no such thing as intentionless language; the only real issue is whether computers are capable of intentions. However this issue may be decided — and our example offers no help in deciding it — the decision will not rest on a theory of meaning but on a judgment as to whether computers can be intentional agents. This is not to deny that a great deal — morally, legally, and politically — might depend on such judgments. But no degree of practical importance will give these judgments theoretical force.” AT p.729 (emphasis added)
By their own account, Knapp and Michaels do not attempt to answer the very intriguing question they pose. Nonetheless, that question has generated renewed interest with the advent of Generative AI chatbots and in view of a provocative scientific paper published two years ago about LLMs by Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell (using the pseudonym “Shmargaret Shmitchell”), entitled “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜”. (“Stochastic Parrots”). Stochastic Parrots raises questions about the safety and utility of LLMs and happens to charge them with lacking intention:
“Text generated by an [L]LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. * * * Contrary to how it may seem when we observe its output, an [L]LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.” Stochastic Parrots at 616–617.
This is the point of departure for the Critical Inquiry online forum, as is nicely presented by its convener, Michael Kirschenbaum. Participants in the forum — including Knapp and Michaels themselves — wrestle with the question of whether computing systems, particularly LLM-based chatbots, exhibit intention. Many do so in reference to the wave poem thought experiment.
The question of intention in computing systems that produce language
Per Knapp and Michaels’ view (at least in AT itself), if computing systems cannot be said to exhibit intentions, then any language produced by such systems would fail to carry any meaning. This theoretically interesting proposition has little to no bearing in the contemporary world. Computing systems both convey and generate language in a variety of settings and applications, and that language carries meaning for those who encounter it. Most of the language presented by modern, online computing systems is conveyed, meaning it is a reproduction of language created by humans. When Google returns a search result, it is typically in the form of a link to a Web site that was composed and published by human beings. Those humans intended the language they used to communicate certain meanings, even if they could not anticipate all of the contexts in which a reader might consume such language or how that context might alter the reader’s understanding of the language. However, with the advent of more sophisticated chatbots and services made available to the public based on LLMs, we are now contending with the reality that an increasing share of language we encounter online will have been generated by one or more computing systems. This is true whether we are directly interacting with an LLM-based chatbot or are encountering text posted on a web site that had previously been outputted from a chatbot. It seems perfectly clear that we can make sense of and find meaning in the language produced by chatbots. How, then, do we square this with both Knapp and Michaels’ central argument and the question they left behind in 1982? That is, if computing systems are now capable of generating language not previously composed by a human author, can we say such generated language has any meaning? In turn, if we can say such generated language has meaning, then must we presuppose that the generating computing system producing it is exhibiting intention? As I argue immediately below, the answer to both of the prior two questions is “yes.” Knapp and Michaels would disagree — and I believe are wrong — for reasons I take up more fully afterward.
Knapp and Michaels do not provide a particular definition of intention in AT. While they raise the prospect that a computer may be able to exhibit intention, their argument throughout the essay is that speech acts and other utterances have meaning only to the extent there is human intention behind them. That human intention must be direct and specific. Direct in the sense that a particular person conveying meaning using language must be the one creating the speech act or language instance in concern. Specific in that the particular language in concern was chosen by its speaker or author to convey a particular meaning. By these criteria, it would be difficult for us to characterize LLM-based chatbots as exhibiting “intent,” based on what we know about how they generate language. Such chatbots produce language based on a sequence of probabilistic inferences — they mathematically determine, step-by-step, which words should be articulated in a sequence to form sentences, paragraphs and the ideas these convey in reference to an inquiry or prompt. If this were our stopping point, we might agree with Knapp and Michaels that chatbots don’t exhibit intent and be left with a dilemma regarding how exactly we do make sense of chatbot outputs. But what if intent — in the context of a digital, highly distributed, and multilayered computing system — exists in fragmented form and can be assembled just in time for a generated output? What if intent could be created as a function of software programming and data retrieval algorithms?
The LLM-based applications with which we are now familiar do not operate on their own, but are designed to react and respond to prompts, or human generated inquiries. The applications process such prompts — and return outputs — based on having been trained on large corpora of data and on being operated using powerful microprocessors in highly distributed computing systems using sophisticated data stores and algorithmic frameworks. There are many layers of natural language processing (NLP) for both prompts and outputs, each one designed and programmed to create the best fit between the prompt and responsive generated output. Many NLP technologies used to run LLM-based applications, such as query processing, were first developed or significantly enhanced in the context of large scale web search applications. Query processing includes methods such as query disambiguation, the goal of which is to determine the intent of the person entering the query, and converting the query to a form most likely to elicit a relevant response from a search-optimized database. In this way the intent of a user is algorithmically determined and the set of results or responses is tailored to match that intent. In contemporary LLM-based systems, beyond relevance and accuracy of an output, things like response tone, sophistication and length are all determined based both on prior programming and, dynamically, on the content of a prompt and the determinable context of its presentation to the LLM-based application.
Each of those facets help create and manifest the intent imbued in a particular output — such intent to be as responsive to the prompter and the thrust of the prompt as possible. In a dialectical setting with an LLM-based application, one can argue that these many layers of computational processing are akin to how the human brain undertakes cognitive processing; LLMs rely on neural network models for processing prompts and outputs and these are functionally analogous to how the human brain works. When in dialogue with other human beings, we must listen to words being written or spoken, interpret what they mean both intrinsically and in the context of the exchange, determine an appropriate response based on our experiences, personalities, moods and memories, and articulate that response word by word. This analogousness does not mean or require that LLM’s exhibit consciousness or sentience in the way that human minds do. But, mechanically speaking, LLMs process language — the meaning of words and phrases, the intent behind a particular prompt and the likely responsiveness to a prompt of probabilistically generated output content — in much the same way the human brain does. If one accepts that intent can be mechanically constructed and executed in a computational system — without consciousness or sentience being a prerequisite — then we can readily accept that outputs generated in response to a prompt in an LLM-based application can and do manifest such mechanically constructed intention. LLM-based systems are machines, not persons. But their magic and unique ability to delight and at times disturb us is based on the dynamic ways they are capable of executing mechanical processes to produce situationally specific outputs. They create the semblance of specific intent because they are programmed to translate general intent (“be a smart and responsive conversationalist”), piece-by-piece and layer-by-layer into generated but non-random outputs responsive to a prompt (“summarize Ray Kurzweil’s ‘The Singularity is Near’ for me in dynamic pentameter”).
On player pianos and the mechanical intent manifested in any programmed system
For illustrative purposes, let us consider the case of a player piano, a device first invented in the late 19th Century. These are pianos that play themselves without a human operator. As devices driven by pneumatics or electro-mechanical mechanisms, the songs they can play are pre-programmed. (In a conventional player piano, the “program” is contained on a physical roll of flexible material having spaced perforations that cause the striking of piano strings when the roll is moving at a defined speed over a metal drum. Each perforation allows air into a small chamber in the drum, changing the pressure in the chamber and producing the mechanical force that causes a hammer to strike the string corresponding to a particular note.) If you are standing in a room adjacent to another with a player piano, you may overhear it playing and assume a normal piano being played by a human being. Even without there being a direct human intent to render the notes being played, you would still hear and understand that music was being played, particularly if you were familiar with the tune. In this situation, we can say there is no intention required, because there is no human playing live or human recording being played back. Here, we would mean that there is no direct, specific human intent required in the moment of playback. Alternatively, and more meaningfully, we can say there was intention manifested in the creation of the programmed piano roll and that intention to render the particular song or music embodied in the roll was what gave the performance meaning and comprehensibility. For this alternative take, imagine that the perforations could be dynamically added to or removed from the strip in response to a listener pushing a button on the piano to request a jazzy version of a particular song or a classical version of that song. The player piano is able to render variants of the song based on its capability to be prompted to switch between “jazzy” or “classical” playing modes, and to patch or add perforations accordingly in real time.
This fancy version of a player piano is what we currently have with LLM-based chatbots. Their seeming ability to improvise really strikes us as exhibiting human-like capabilities, including intent. But just as the appearance of being human-like in the production of song style variants would render the piano no less mechanical in its operations, the appearance of being human-like in the production of conversational, highly responsive speech renders LLM-based chatbots no less mechanical in their operations. In both instances, we can say more sophisticated methods for programming intent into the system have been accomplished. Or that no direct, specific human intent is required in the first instance for the systems’ outputs to be comprehensible in the way that similar output created by a human being would be. All mechanical systems that are programmable to perform specific functions, particularly emulating one or multiple human capabilities, are the non-accidental creations of human beings. Execution of that programming to accomplish those specific functions always manifests human intent, however attenuated or fragmented.
Why Knapp and Michaels’ position in Against Theory does not permit a straightforward explanation of the meaning we apprehend in Generative AI chatbot outputs
Let’s revisit the two questions I posed earlier to tee up discussion of my position:
“[I]f computing systems are now capable of generating language not previously composed by a human author, can we say such generated language has any meaning? In turn, if we can say such generated language has meaning, then must we presuppose that the generating computing system producing it is exhibiting intention?”
Per Knapp and Michaels, our answer to the first question is yes only to the extent our answer to the second question is yes. That is, generated language has meaning only to the extent the generating computing system exhibits intentionality. Within the bounds of Knapp and Michaels’ reasoning, we would thus appear to have a dilemma on our hands. Either our experience of generated outputs is mistaken, and the appearance of intentionality is illusory, or, though mathematically determined, LLM-based computing systems are exhibiting a form of intention with which we must reckon. The first possibility is inconsistent with Knapp and Michaels’ views in AT; we can’t have both illusory intention and meaning in generated language. The second possibility may be consistent with their views, but it is highly doubtful that Knapp and Michaels would view the kind of intention being exhibited by LLM-based computing systems as legitimate or proximate enough to what they conceive of as “intention” in AT. In fact, years later, they express skepticism in their recent Critical Inquiry forum piece that language produced by machines is any different than language produced accidentally in nature. They conclude their piece by asserting “no one really thinks the texts that AIs currently produce are meaningful and that everyone continues to — can’t help but — act on the assumption that what a text means and what its author intends are the same.” As such, Knapp and Michaels seem to leave themselves no way out: LLM-based chatbots don’t exhibit intention and so the language those chatbots produce don’t — and can’t — carry any true meaning.
Knapp and Michaels’ explanation of what is going on comes down to a chatbot user stipulating intention: “[I]n order for us to use [an LLM’s outputs produced by an algorithm], we need to treat it as if it were producing a speech act — in effect stipulating that the AI means what an ordinary speaker of English might mean if she produced those marks and spaces.” Effectually, a user of a chatbot is creating the fiction that the LLM’s outputs are imbued with intention. Intention doesn’t come from the act of the chatbot creating language but from what the reader of that language supposes to be intentional speech. This recent conclusion would appear to violate the more straightforward argument they made in AT:
“[T[he relation between meaning and intention or, in slightly different terms, between language and speech acts is such that intention can neither be added nor subtracted. Intention cannot be added to or subtracted from meaning because meanings are always intentional; intention cannot be added to or subtracted from language because language consists of speech acts, which are also always intentional. Since language has intention already built into it, no recommendation about what to do with intention has any bearing on the question of how to interpret any utterance or text.” AT at 736.
If intention cannot be added to or subtracted from meaning, how is it that chatbot users get to “stipulate” meaning — and therefore intention — in chatbot outputs? Moreover, why, instead of interpreting the direct and specific intention of a particular human in AI outputs, do we get to stipulate, generically, that “the AI means what an ordinary speaker of English might mean if she produced those marks and spaces”? As I noted above, the bind that Knapp and Michaels get themselves in arises from their implicit — but undefined — notion of “intention.” In AT they offer no formal definition of intention and appear to take for granted that their readers have a common understanding of what that term means. Their descriptions of authorial intent in AT appear to presume a single human author behind any given text or language utterance. This is an overly limited notion of intention based on what we know about computers and how they function. Computing systems are not accidents, but very much intentional creations of human beings, and therefore the language they produce is also not accidental. Once we recognize this, we can break free of a false dualism: the idea that (1) language either must have a human author and thereby specific, direct intent in order to be meaningful, or else (2) such language is accidental, thereby devoid of meaning, and not worth interpreting.
Other than Knapp and Michaels themselves (who attempt, unsuccessfully, to preserve their original argument in AT), the commentators participating in the Critical Inquiry forum acknowledge in different ways the reality that we can and do find meaning in computer generated text. It is silly to pretend that computer generated text does not convey meaning and that we should ignore that text altogether. However, while no commentators offer a very coherent explanation as to why this is the case (the forum imposed word limits on submissions!), most offer some pertinent insights. Seth Perlow’s piece suggests that much of human expression is machine-like, and so our ability to understand language produced by machines is non-problematic. While not central to the argument he makes (which is largely focused on refuting Knapp and Michaels original position), he offers this observation, with which I wholeheartedly agree: “Antitheorists might respond that LLMs do reflect human intentions — those of their programmers, their users, and the authors of their training texts — but these intentions become highly attenuated in the textual details of their outputs.” Reflecting on the rise and ubiquity of computers in our lives over the past 40 years, Alex Gil suggests “[s]omething happened to our use and usage of meaning and intention along the way, and our own time is asking us, begging us, to reformulate our theories of language and meaning, sans intention.” Kari Kraus notes that Stochastic Parrots suggests “no redemptive comparison between human and synthetic language.” She concludes: “The solution is not to use meaning/intention as a wedge to deepen the divide between humans and AI but instead to develop new, more expansive theories of meaning that recognize continuities as well as discontinuities between their respective domains.” Hans Bajohr’s piece perceptively states:
“Intention, thus, can come in a fictitious modality, operating with shades of meaning, and be neither just illusory pareidolia nor the mistaken belief in sentient machines. Rather, such a fiction is a perfectly legitimate pragmatic prerequisite for successfully interacting with machines. By insisting on meaning as binary and coeval with intention, Knapp and Michaels cannot account for what is already happening; and so we need theory.”
Finally, Katherine Hayles in her afterword to the forum, appears most attuned to the misfit between Knapp and Michaels’ view of intention and the realities that we now face in the age of Generative AI. I reproduce her conclusion here as it is about as on point and consistent with my position as I could have surmised before reading it:
“The question, as I see it, is not whether these texts have meanings, but rather what kinds of meanings they can and do have. In my view, this should be determined not through abstract arguments, but rather through actually reading the texts and using an informed background knowledge of LLM architectures to interpret and understand them. The proof is in the pudding; they will certainly elicit meanings from readers, and they will act in the world of verbal performances that have real-world consequences and implications. The worst thing we can do is dismiss them as meaningless, when they increasingly influence and determine how real-world systems work. The better path is to use them to understand how algorithmic meanings are constructed and how they interact with other verbal performances to create the linguistic universe of significations, which is no longer only for or of humans.”
Copyright © 2023 Duane R. Valz. Published here under a Creative Commons Attribution-NonCommercial 4.0 International License
The author works in the field of machine learning/artificial intelligence. The views expressed herein are his own and do not reflect any positions or perspectives of current or former employers.
