A Portrait of the Artist as a Young AI: Testing the Narrative Limits of GPT-2

OpenAI has just released the full version of GPT-2 for anyone to explore.

With 1.5 billion parameters, this AI is by far the most powerful language model I’ve ever seen, trained on eight million web pages and capable of generating human-sounding text in many different styles.

You can experience the model for yourself at Talk To Transformer, a website created by Adam King that runs the full-powered GPT-2 model for anyone to sample. Just feed the AI any text prompt, and it will generate a response that could easily pass for a piece of human writing.

For instance, I typed “When will GPT-2 write its first novel?” into the Talk To Transformer site and got this completely computer-generated response that outlined a sci-fi fantasy adventure and a Kickstarter campaign.

Watching GPT-2 effortlessly produce six mostly coherent paragraphs in a few seconds, it’s easy to imagine this tool being used to write entire novels someday.

While working on a story about National Novel Generation Month for Publishers Weekly, I interviewed Robin Sloan, author of Mr. Penumbra’s 24-Hour Bookstore and Sourdough.

“Oh my god, are robots gonna take over the job of novelists? I think that’s basically a nonsensical question,” Robin Sloan told me in a quote that didn’t get included in the final article.

In his workshop, Sloan runs a version of GPT-2 on GPUs he purchased from a Bitcoin miner. While he uses this language model as a writing tool, Sloan isn’t worried about getting replaced by AI:

“It’s just obvious that there’s not a shortage of really good novels written by humans. All things being equal, people would rather read something written by a person, with the point of view of a person, about that person’s existence in the world.”

A Writing Tool, Not A Writer Replacement

That said, GPT-2 can help writers explore new territory, working together to create something new and wonderful.

Sloan has used GPT-2 as a writing tool, generating short, customized fantasy stories for his newsletter readers. Each customized story got mailed to Sloan’s readers as a special gift.

You can read about the project here. One happy reader posted a screenshot of the fantasy story they received:

So what if we use GPT-2 for writing inspiration? I’ve been tinkering with smaller models of GPT-2 on my own laptop, learning how to “fine-tune” the AI to tackle specific kinds of writing.

I’m taking the NaNoGenMo challenge this month, attempting to generate a book inspired by my favorite subreddit, Writing Prompts. When finished, my NaNoGenMo manuscript will consist of a series of AI-generated writing prompts and AI-generated responses to those writing prompts.

As I worked on the project this week, I’ve realized two things.

1- AI writing will require lots of human interaction. Coders, editors & readers must sculpt AIs potentially infinite output into something worth reading.

For my project, I fine-tuned my Writing Prompt Prompter AI on thousands of sample writing prompts. I gave that AI a little guidance, asking the model for a writing prompt set “in a world where AI writes all the novels.”

After sifting through 50 samples from my fine-tuned GPT-2, I found ONE computer-generated writing prompt that had a wonderful singularity:

[WP] In a world where AI writes all the novels and stories, poets stutter-step before their fights and have to fight for their poetry. You are the first one to find a better way than anyone else and bring it home with you as a trophy. Your name is Mr. Rogers.

2- If you give the fine-tuned GPT-2 model a sculpted set of prompts, you can generate small-scale narratives that actually tell a story.

I also created Writing Prompt Responder, an AI fine-tuned on thousands of examples of great writing prompt responses. I fed that “ In a world where AI writes all the novels and stories…” writing prompt into Writing Prompt Responder, generating 30 different writing samples.

Most of those samples were more or less coherent on a sentence level, but hilariously inept or downright incoherent as a complete story. However, in one sample, my fine-tuned Writing Prompt Responder created an entire story that unfolded over four paragraphs.

It told the story of an aspiring poet, his relationship with his parents, his poetry training, and ultimately, his decision to pursue his dreams (despite the adversary he faced along the way). A Portrait of the Artist as a Young AI, if you will.

While this is no great work of literature, it was fascinating to watch the fine-tuned AI create a sense of narrative coherence within the small structural constraints of a writing prompt response.

Feel free to suggest new prompts and new themes to tackle with my two AIs.

Here’s the entire computer-generated story…

I had asked my parents for advice. They were at odds over whether I should bother writing or not, but my father insisted I should go. My mother insisted I just keep at it, and by the end of the week, she’d entrusted me with writing materials, specifically scribes’ notes for the upcoming poetry competition.

I grumbled at first, but started to take action anyway. I asked my father for some of his old writing supplies, and he happily lent them to me. From that point forward, I took to spending my days in the studio, perfecting my craft. I took on writing duties the day the poems were due, and began filling in the blanks with my own thoughts and reflections. I showed them not just the poems, but also full chapters if needed. My father even helped me with that, sketching out my steps and plot arc from beginning to end.

However, as the weeks passed by, it became increasingly difficult to stay focused on my goal. My steps grew more sporadic and numerous, and my mind drifted. I wondered what was happening in the world, who was interested in poetry, and how I was going to get them all interested in poetry. I turned to alcohol, and found my steps more painful and difficult to imitate. I would much rather spend my time in bed, furiously erasing the steps of those around me. I turned to illegal drugs, and found that instead of relaxing and reflecting, my thoughts grew more chaotic, and my actions more destructive.

By the end of the week, when the judges had arrived to hear the poem written by me, I was left with a bittersweet smile on my face. I was proud of myself for taking the challenge, and now that I was going to university for my English, I wanted to continue this legacy and become a renowned poet.