When an AI Writes Wikipedia
How much does GPT-Neo know about us?
Jonathan Swift’s 1726 novel Gulliver’s Travels describes a ‘wonderful machine’ that permits “the most ignorant person, at a reasonable charge, and with a little bodily labour” to “write books in philosophy, poetry, politics, laws, mathematics, and theology, without the least assistance from genius.”
Imagine my surprise and delight when, nearly 300 years later, I stumbled across EleutherAI’s version of this very machine: GPT-Neo. With a few clicks, I was able to generate hundreds of thousands of words. More specifically, I was able to create Wikipedia-style biographies for the 118 Nobel Laureates in Literature without the hassle of research, the despair of a blank page, or the need, really, to labor at all.
I discovered the GPT-Neo’s ability to generate Wikipedia pages accidentally. Simply typing a first and last name followed by life dates — for example, “Doris May Lessing (22 October 1919–17 November 2013)” — was all it needed to go forth and create an entire Wikipedia-like biography, often complete with a list of categories like those found at the bottom of real Wikipedia articles–‘American women novelists’ or ‘Accidental deaths in Mississippi’, a category to which William Faulkner belongs.
My challenge was not writing, but reading these pages, which I did. And I discovered that GPT-Neo can be a bullshitter in the true academic sense of the word:
“A liar is someone who is interested in the truth, knows it, and deliberately misrepresents it. In contrast, a bullshitter has no concern for the truth and does not know or care what is true or is not.” [Ian P. McCarthy et. al]
Still, GPT-Neo managed to capture the feel of a Wikipedia page quite well. The 1.3B (1.3 billion parameters) version identified 43 of the literature laureates as authors, and nine as Nobel prize winning ones, while the 2.7B version identified 51 of the laureates as writers (including most, but not all, of the same people the 1.3B version identified). As I read these biographies, I began to wonder how much GPT-Neo ‘knew’ about the individuals it so effortlessly described.
“Her most famous work is The Bluest Eye, which was a major part of her career,” GPT-Neo wrote of Toni Morrison. One might disagree that The Bluest Eye is Morrison’s most famous novel, but she did write it. Of Hemingway, the machine noted that “his later works, such as The Sun Also Rises (1924), The Old Man & The Sea (1925), Across the River and into the Trees (1933), and The Garden of Eden (1939), are usually described as “historical novels.” The pub dates are off, but the titles are Hemingway’s.
Such accuracy was, however, the exception. In general the mechanical writer didn’t care if it got the details right:
“Thomas Mann was the author of twelve books, including The Magic Mountain, The Magic of Truth, The Magic of Reality, The Magic of Inflation and A Man Without Qualities,” it wrote.
It confidently attributed works to whomever it fancied: “Faulkner’s most famous works are The Sound and the Fury, The Good, the Bad, and the Ugly, and The Hamlet Tree and Other Stories,” while Alice Munro’s oeuvre included “The Girl Who Leapt Through Time…followed by My Life Is a Lie, The World That I Love, and The End of the Rainbow.”
Awards and ‘firsts’ featured prominently. “In 1958, after the birth of their son, [José Saramago] was declared the Father of the Year by the Portuguese Academy of Letters” or “a few months after his death, [Miguel Angel Asturias] was ranked #24 on People magazine’s list of 50 Greatest Spanish Actors.”
Sometimes, these claims contained footnotes: “[Sinclair] Lewis was born on February 7, 1885 in New York City, New York, the seventh of nine children, and the eleventh out of twelve children in a large family.[1]” But they were footnotes to nowhere. Other times, the mechanical writer generated an ‘External links’ section containing links to author interviews or books, which I didn’t find when I searched the internet.
Overall, GPT-Neo’s inventions–though sometimes comical and usually untrue–had an eerie consistency that made them convincing. The subject’s life span and life events almost always matched the dates provided. GPT-Neo simply fabricated the story in between birth and death, weaving real people into its narratives– “In 1914, [Nelly Sachs] met the German-Jewish anarchist Rosa Luxemburg,” or “Beckett was influenced by the work of Henry James” — and generating quotes to support the subject’s notability: “The American poet laureate, Charles Wright, once described Pinter in the New Yorker as ‘one of the most important writers in the English-speaking world.’”
Sometimes, the authors themselves spoke: “In a 1986 interview, [Bob] Dylan said ‘I have a thing for girls who do not dance. I am a dancer with a problem.’”
Nowhere was GPT-Neo’s bullshitting skill more apparent than in its lengthy passages describing alleged philosophical work:
According to Eucken, the human will can be understood in three ways: as a pure subject determined by the will of the will of God; as a will which does not have a will; as a will which is determined by an image of the will of a superior…. The will of the will without will is a pure psychological phenomenon which is the image of this will. The human person is the subject of the will of God, but this does not mean that the person has a complete personality.
GPT-Neo wrote about “the problem of the limits of knowledge and the concept of truth,” “concepts of freedom and determinism,” and “the question of whether the concept of truth is really just a matter of opinion.”
How, you might be thinking, did it learn all this?
GPT-Neo was trained on The Pile–a huge text data set that includes the English Wikipedia, among many other corpora. It comes with this disclaimer: “known to contain profanity, lewd, and otherwise abrasive language. Depending on your usecase GPT-Neo may produce socially unacceptable text.”
Was the generated text unacceptable? At times, yes. Was I reminded of Wikipedia’s known biases? Also, yes. The work of Sigrid Undset, who is described as a “translator, journalist, and editor” is honored in this way:
Several items related to her life are being displayed at the Museum of Art and Literature in Uppsala together with other memorabilia from her life. These include her wedding dress, the family portraits, letters from her husband, and some personal items.
Louise Glück’s biography begins by noting that she is “a German chemist and professor of chemistry. She is the daughter of Ernst Heinrich Glück” before spinning into a multi-paragraph digression on the accomplishments of Ernst Heinrich. The text eventually returns to her work, but the space that should have gone to describing her accomplishments was filled by her family, and in this case the work of a man.
Did the mechanical writer reference men disproportionately? Did it use different adjectives to describe men and women, or dedicate more space to the personal lives of the female subjects? I’m not sure. My sample was small. Of the set of the 118 Nobel laureates I looked at, only 14 are women. However, I would not be surprised if the mechanical writer perpetuates or even amplifies the known structural biases of the human-generated Wikipedia.
Overall, I found GPT-Neo’s writing lucid, accurate at times, false far more often than not, and entirely without citation. My role in assessing the biographies became a daunting one of fact checking, and I shudder to think how convincing the text might appear to an unsuspecting reader if the mechanical writer chose also to fabricate a bibliography to ‘prove’ its claims.
As I reader, I do not trust the mechanical writer. But my feelings mean nothing to this ‘wonderful machine,’ which will happily generate text without the need for “bodily labour” or even a body at all.
The computer generated biographies are intriguing, unsettling, and at times quite funny. I put together this collection of the opening lines of each to give a sense of the work.
Find out more:
Sid Black, Leo Gao, Phil Wang, Connor Leahy, Stella Biderman,
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow (2021), Zenodo
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy, The Pile: An 800GB Dataset of Diverse Text for Language Modeling (2020), arXiv.org