What in ‘tarnation’ — How I dealt with ChatGPT hallucinations
I started writing Twice Removed shortly after reading Nick Bostrom’s Superintelligence in late 2015 (a must read for anyone remotely interested in ChatGPT). The thought experiments in that book were fascinating but one wouldn’t let go of my mind: the paperclip maximizer. It is a ridiculous, runaway thought experiment and at the same time an unsettlingly plausible scenario (the concept, not the paperclips).
It goes like this: assume humanity faces an existential problem. This problem is so significant that it seems a worthy gamble to let an AI (general artificial intelligence) take a shot at solving it without restraints. Computing power paired with manufacturing, resource allocation, and decision making ability are given to the AI with one task assigned: find the solution. We’ve always kept restraints on AI for good reason, but this problem is going to destroy the world/society anyways so the people in charge decide it’s worth the roll of the dice.
Civilizations arrive at this point of desperation far too much to say this isn’t a plausible scenario. Anyone that lived through the Cold War can attest that the entire concept of mutually assured destruction was a daily game of chicken and dice rolled into one. In modern times we are witnessing the earnest discussion and action of people trying to spread particles in the atmosphere to reflect more sunlight as a way to slow global warming.
In Superintelligence, truth is said in jest by tasking the AI with finding the single best method for creating paperclips. The goal is efficiency and scale. Without proper parameters in place, the AI was able to find ruthless efficiency toward its goal with no defined stopping point. As instructed, the AI starts by improving factories before moving on to mining methods, energy grid priorities, and finally the planets of the solar system to carry out its goal. Another scenario posits an AI tasked with creating the most efficient energy grid and it covers every square inch of the planet with solar panels. Good AI, poor instructions.
Any intelligence has this problem of undefined goals. You can see this behavior with predators like wolves that kill without consuming their kill — sometimes the means can overpower the ends. If you’ve ever been a manager, a parent, or a scientist in charge of graduate students, you likely have your own memory of tasking someone with a project only to find days later that they have taken the task far beyond the appropriate stopping point.
I distinctly remember an instance from early in my research career where I went too far down a particular road trying to reach the goal assigned by the head of my genetics lab at Washington University in St. Louis. I was tasked with creating a unique way to immortalize certain samples in liquid nitrogen for later use. After all the research, experimental design, execution, failure, redesign, and about 5 loops later I had a checkin with my PI where he just shook his head.
Luckily, he caught me before I put too many resources toward finding the solution but at its core this was a paperclip maximizer problem. He tasked me with an unbound goal and set me to work but what he really wanted was for me to test the low hanging fruit of possibilities. The subtle boundary that he could recognize wasn’t visible to a less experienced scientist like myself. Fortunately, he was a fantastic mentor that shared his decision making process over some beers.
In silico, unclear boundaries can make a coding loop go on forever. Computers are (were?) dumb and will execute the exact command provided. When problems aren’t anticipated in a game this will cause the game to crash. Patches solve these problems. In the stock market this can manifest weird feedback loops that quickly alter the market but at the end of the day the market will shut down if behavior strays too far from the norm. On something with more immediate consequences like complete control of the supply chain or autonomous vehicles in the search for efficiency, these unclosed loops can create disaster.
This behavior fuels the fear of AI: even the best of intentions can go awry simply because it’s impossible to anticipate everything. If you want a taste of this complexity in a low stakes setting, you should play the game Factorio.
After writing Twice Removed in 2015 I took some time away from the story to focus on an unrelated novel. Editing Twice Removed was getting in the way of the creative process and I didn’t know how long that muse would be talking to me for my new work. Life happened and years passed without editing either of these books.
Then came the large language models like ChatGPT. After marveling at the parlor tricks like image creation and writing poems, I discovered the immense power of ChatGPT as a coding assistant and editor. Twice Removed was edited for grammar within 2 hours start to finish with a complete list of changes and justifications for each change. If something didn’t make sense of the intentional dialect of a character was changed it only required a simple back and forth with ChatGPT to fix it. It was a blissfully simple experience… until I started finding hallucinations hidden in the work. Some of the ways it changed the message was semantical like a changing fast to quick. These were rare but a clear fragment of this statistically driven large language model.
The most significant change occurred during chapter two. After an excellent editing job, ChatGPT added three unique paragraphs to the end. These paragraphs continued the logical next steps of the plot that were intentionally left out because they aren’t significant to the plot. Telling your friend about a kayaking trip doesn’t require the part about filling the gas tank or putting on sunscreen. ChatGPT created these types of mundane parts of the story out of nothing more than the chapter’s plot to go on. Ok, so what if the ChatGPT added some mundane parts?
Tarnation. That was the word. Even though it fit the character’s voice, it is a word I would never use. It’s one of those words that appear in films or written works; tarnation is a word that never never seems to come out of a real person’s mouth. I’ve lived for years in parts of the country where you would expect to hear ‘tarnation’ but it never once came up. This word has been the subject of a few late night debates in my time — I would never, ever use it.
But there it was. Tarnation was there on the screen in plain sight like a snake in the grass. Then the body of the paragraph came into view.
The interaction between characters made sense, but with each passing exchange they sounded less inspired, less real. Like the slight differences of intention between two different inflections on the word “yes.” A tense situation between two characters quickly dissolved into conflict de-escalation techniques that would be coached by a counselor.
The final AI-generated sentence of these hallucinatory paragraphs reads “They both hugged him, the three of them standing in the driveway for a moment, caught between the real world and the virtual one.” This sentence has kept me up at night. This interaction could easily happen in this story, but in this moment the real-virtual link simply doesn’t make sense. Furthermore, the virtual world in Twice Removed wasn’t even mentioned in this chapter or editing thread.
At a certain point I revised the workflow and only edited smaller chunks of prose at a time. This helped ChatGPT stay on task and limited the ways it would spiral toward its own interpretation of the story. Twice Removed edited with an AI assistant could turn out limitless versions of the same story with minor changes here and there. What is presented in this version of Twice Removed is the truest version of my original vision with only editing grammar and syntax. The fact that I had to wrestle with ChatGPT to stay on task and operate within clearly defined boundaries only speaks to the problems explored in this story.
This has been a fascinating process. In some ways it brings the story to life in a manner that I would have never anticipated. Are languange models perfect yet? No, but they are getting better by the day. For tedious tasks, they are already more than sufficient when compared to working with a tired or hungry or incompetent human that needs to be scheduled weeks in advance for simple answers to disparate or jargon filled questions. It feels more and more that we are exiting the woods and taking our first steps across the uncanny valley.
If you would like to read Twice Removed, feel free to follow along here on Medium. I will be putting out one chapter per week.
