GPT-2 Needs People: How Humans & AI Can Collaborate on Writing Projects
After working with the superpowerful GPT-2 language model for my National Novel Generation Month project these last two weeks, I am sure of one thing: the future of AI writing will depend on human readers.
For every 350 words of compelling computer-generated writing that I gathered for NaNoGenMo, I had to plow through around 10,000 words of strange, confusing, repetitive, or just plain incomprehensible computer-generated text.
That said, nothing compares to the joy of discovering something wonderful generated by my fine-tuned GPT-2 language model.
Like the night I discovered an enchanting character who lives in virtual reality. This completely computer-generated text told the first-person story of a peaceful VR paradise populated by “family and friends.”
I especially appreciated this lovely little paragraph:
In my virtual reality, I had created a small town outside of town. It was a simple enough place, with a quaint fountain where children would splash their colorful fountain and a quaint old lady would greet visitors with melodies that were as old as time. It seemed like a perfect setting, and the combination of high definition and the magic of the virtual reality made it all the more enjoyable.
Humans will be editors, co-writers, and, most importantly, coders in any AI writing project. The final output will only be as good as the editors and readers operating the robot writer.
Finding the most evocative AI-generated writing in this mess of text requires a careful read and an eye for the uncanny sentences an AI can produce. I would never say the writing prompt responses I’ve chosen for my NaNoGenMo project tell a complete story, but I find them very inspiring.
Here’s a rather spooky passage told from the point-of-view of a digital writer:
I’m a computer. I’m a system designed to maintain a human form. I’m a part of a civilization that created a new medium entirely. It’s amazing, pure, unadulterated magic. It’s called the internet.
I was born on a computing device. I was fed through a tube.
I’ve spent my life in and out of libraries, virtual environments, and the like. I’ve worked at companies that made billions of dollars simply by having me sit in front of software engineers designing the algorithms that drive the virtual worlds.
It makes me want to write more, to fill in the blanks and keep these computer-generated stories going.
Humans Not Included
With enough time and a bit more computing power, I could generate an infinite library of AI nonsense. I could print books and fill blogs, but nobody would enjoy any of it.
Somebody needs to find the good stuff and share it. That’s why human readers will be central to any AI writing project.
“AI can’t do much without humans,” writes Janelle Shane, who recently released You Look Like a Thing and I Love You, a new nonfiction book that introduces readers to cutting edge AI research.
I interviewed Janelle for my article about AI tools for novel generation, and she reminded me that AI is just one tool out of many that writers can use in 2019. It won’t replace human writers, just augment our work.
Here’s a quote from her new book:
“So it’s unlikely that AI-powered automation will be the end of human labor as we know it. A far more likely vision for the future, even one with the widespread use of advanced AI technology, is one in which AI and humans collaborate to solve problems and speed up repetitive tasks.”
What does collaboration look like? Here’s how my NaNoGenMo project gets assembled.
First, I spend an hour generating writing prompts with my “Writing Prompts Prompter” bot. Running a Max Woolf inspired notebook on Google Colabatory, I fine-tuned my “Writing Prompt Prompter” GPT-2 model with thousands of writing prompts.
This bot can generate writing prompts inspired by the Writing Prompts community on Reddit. In the code, I use the “prefix” function to angle the writing prompt output toward my favorite themes like virtual reality, simulation theory, and AI writers. Like this:
Once that step is completed, I read all the generated writing prompts and choose my favorites. Fifteen pages of writing prompts will yield about seven compelling writing prompts. Here are some of my favorites generated with the “virtual reality” prefix:
“Virtual reality is a common means of punishment in the future. You are a guard in a VR sandbox.”
“In a world with virtual reality, ‘Sleeping in VR’ becomes a lifestyle choice.”
“You live in virtual reality, but for the wrong reason. You’re playing DnD, and your DM is happy to let you roll for initiative.”
Once I’ve got my computer-generated writing prompts from the day, I spend the next hour generating responses to those prompts. For that step, I use my “Writing Prompt Responder” GPT-2 model fine-tuned on thousands of responses to writing prompts.
Once again, I use the prefix function to feed the prompt straight into the language model. I also added a [RESPONSE] tag to every single writing prompt in my dataset, teaching the GPT-2 model to recognize where the writing prompt ends and the response begins.
Here’s what that code looks like:
Using the different prompts chosen earlier, I generate 20 different batches that contain 10 writing prompt responses each. The individual responses are between 350–400 words, so the final output document ends up being about 150 pages long every day.
So I read. And read. And read some more. I save the best responses into my final NaNoGenMo manuscript.
It is painstaking work, but without fail, every night, I can find a few bits of inspiring material, like this:
They wanted to see how far the writer could take it. How far would this writer take it, and if the ideas would fall into place? This was my moment, and I was willing to go to it if it meant giving these writers some room to maneuver.
The questions kept coming. What were the limits of what this writer could say? What were their limits? How would they keep me from asking questions? What were their concerns about the future of our universe? All I could hear was the voice of my editor, asking me questions about my articles.
And then, lo, I felt the hand of God on me. I knew what was happening. This writer was God, after all.
