Generative AI and the Future of the Web
Imagine a scientist reading a new research paper. The results seem important in the context of her obscure specialty. Routinely, she checks the citation for a key supporting result and discovers a paper she is not familiar with, from a researcher unknown to her. As she continues reading she begins to wonder if her mastery of her chosen specialty has lapsed. She decides to read the other paper next. Then comes the surprise: the other paper doesn’t exist, nor do the authors to whom it is attributed, and she realizes that the paper she just finished reading was fraudulent. Today, cases of scientific fraud are rare enough to seem newsworthy, but the future could be very different. As Zurich-based physicist and data scientist Theresa Kubacka explains in this tweet, large language machine-learning models (LLMs) like ChatGPT are largely capable of automating the production of plausible papers replete with scientific results that appear real but only exist as machine learning hallucinations. Imagine an Internet saturated with plausibly interesting content from non-human sources, some created with an intent to be truthful and useful and some with rather different motives. How will an honest reader survive?
The scenario I describe above is hardly unique to ChatGPT. Meta (formerly Facebook) felt compelled to withdraw their own Galactica LLM from public exposure after only three days, in the wake of searing criticism of the system’s tendency to “reproduce prejudice and assert falsehoods as facts.” In 2021, Sundar Pichai presented Google’s LaMDA model as if it was on-track to become the way we use Web search. While I don’t know of any papers LaMDA has written, it did facilitate a minor scandal when a Google engineer announced his conclusion that LaMDA was sentient. His proclamation was followed shortly thereafter by his dismissal, although LaMDA is still with Google.
Like the Internet, large language models are not inherently safe.
While certain journalists and their more gullible readers may express outrage at the danger LLMs represent for society, such a position reflects a naive and problematic expectation that such technology could be inherently safe. We take for granted the safety of airplanes and automobiles, but accidents make it painfully clear these technologies are not inherently safe. The relative rarity of accidents reflects how society has responded to these technologies with norms and policies to make them safe. Transportation technologies are hardly unique in this regard. Consider electrification, plumbing, healthcare, and even modern food production. All of these technologies have transformed modern society, and none of them are inherently safe. They become safe when society applies norms and standards to make them safe.
Like the Internet, large language models are not inherently safe. Like the Internet, they are a groundbreaking technology with many potential benign and beneficial uses. A broad prohibition of AI-assisted authorship and publishing seems not only unenforceable but could also deny the public of potentially beneficial applications.
Academic publishing uses a system of citations that helps the researcher in the example above determine she is being mislead. The generality of this system makes academic publishing more robust against fraud, regardless of whether that fraud involves an LLM. Professional journalism applies a different but comparable set of norms, including transparency of authorship, editorial staff, conflicts and corrections. These norms provide similar support for the integrity of news publishing. The norms-breaking practices of online publishing means that Internet publishing does not automatically benefit from the integrity guarantees of these legacy media systems. What would it take to fix that?
Imagine a world where the corpus of published work freely includes work produced or assisted by LLMs. How can an information consumer navigate such a corpus safely? It seems unlikely that ChatGPT, Galactica and LaMDA behave identically. Basic differences in how the systems are trained and deployed will impact their suitability for various purposes. An LLM engineered to support scientific research might behave very differently from one built to produce compelling fiction. Over time, as technology adapts to marketable needs, systems will evolve and specialize depending on the skill of their development team and the need they target. In academia, scholars build reputation and credibility through their publication history. LLMs will need a comparable system.
Given the nascent nature of LLM technology, elaborate restrictions on how these systems are engineered or used seem premature at best. An alternative, complementary approach is to consider what information consumers need to use these systems safely. What context should a reader have to enable an informed decision about whether a publication is trustworthy? Lest we mistakenly trivialize this challenge, we should acknowledge that even without LLMs such trust decisions are hardly straightforward, so a goal that consumers have perfect or even adequate information for their trust decisions is unrealistic. A more practical aim is that the introduction of LLMs not make the challenge for a modern information consumer worse.
One detail that seems obviously useful is whether a particular publication was produced with or without an LLM. If a publication was produced without an LLM or other novel technology, then a consumer can apply the same information literacy skills and tools we use today and nominally satisfy our goal of not being less safe. In a world where such technology dependencies were declared and honest, this simple dichotomy of with or without an LLM might be adequate. Unfortunately some publishers will be dishonest, careless, or incompetent. This challenge can be addressed through a common pattern in publishing: accountability. Authors and publishers are commonly expected to be honest about their legal identity and their publishing history. Though accountability, good publishers are rewarded, while dishonest publishers are subject to legal and reputation consequences when their published work is deceptive, dishonest, incompetent or illegal. Taken together, these two dichotomies suggest a simple two-dimensional grid as illustrated in Figure 1 below:
In this diagram, the “legacy publishing” quadrant is roughly comparable to publishing prior to the Internet. “LLM-assisted publishing” corresponds to LLM-assisted publishers who subscribe to norms for identity transparency and archiving that are comparable to legacy publishing. In this quadrant, a consumer knows that a published work used an LLM, that the author and publisher are subject to norms of honest identity and durable content archiving, and that the author and publisher accept responsibility for the integrity of their content. The “UGC publishing” quadrant overlaps with current practice in online publishing, where content respecting traditional publishing norms exists alongside user-generated content (“UGC”) that doesn’t necessarily observe legacy publishing norms. Note that accountability cannot be assumed in this quadrant, making online content less safe than legacy publishing, irregardless of LLMs. This represents a choice by online platforms to prioritize the anonymity of publishers over the public right to know which publishers are accountable, and makes the public less safe relative to legacy publishing.
In the fourth quadrant, labeled “XGC Content”, LLM authorship is implied, but publisher accountability is not. This quadrant represents the most extreme position on free speech, prioritizing anonymous free expression, both for humans and automata, over the public’s right to reliable information about publishers.
By adapting norms and standards from legacy publishing to support these four quadrants, and isolating the corpus of publications that use accountable LLMs, LLM-assisted publishing could approach a level of safety comparable to legacy publishing. A search engine that incorporates these quadrants into its ranking function and exposes them in search results allows users to benefit from safety norms comparable to legacy publishing. Keep in mind however that the details matter. A platform that provides identity verification as a service must be competent, with robust detection of identity fraud and clarity on the difference between organizations and humans. Twitter’s recent example demonstrates how an identity verification service that lacks competence can be attacked.
Accountability norms may not be the only approach to making LLM content safe to search. A content search service might instead attempt to automate detection of LLMs and determination of content quality, foregoing the publisher friction induced by declared identity and archiving. The detection of LLMs would likely be subject to attack by motivated adversaries at scale, creating a significant technical challenge for a search service. Content moderation is also a challenge, as automated moderation is equivalent to automated determination of truth. Given that truth is undecidable by humans, it seems problematic to predicate public safety on an automated solution. While automated content analysis performs well in certain limited contexts, it is hard to scale to languages and search terms that lack adequate coverage in the content corpus. This can lead to problems like “data-voids” that can be exploited by malicious publishers.
Humans are not alone in their need to reliably recognize content from LLMs. Given the propensity of LLMs to produce compelling content that is factually incorrect, an LLM intended to produce factually correct content could benefit from a way to triage training inputs based on LLM authorship and accountability, much like a human. Also, don’t assume that authorship conventions alone are adequate to make an LLM-enabled Web safe. Unless those conventions penetrate casual online communications like email, generative AI will continue to be a handy tool for phishing and other forms of abuse.
Fraud and incompetence are not new phenomena in academic publishing. They are not problems created by machine learning or LLMs. The integrity of academic publishing is built upon a system of norms and standards, developed over centuries of practice, that support the honesty and integrity of participants in the academic community. This system does not depend on participants acting in good faith. Instead, it is based on standards for transparency, attribution and documentation that provide essential context for an author’s claims, facilitating verification of claims and the normal development of reputation and accountability. It is a system that rewards honesty, integrity and competence, while exposing dishonesty and incompetence for the benefit of the community.
The integrity of academic publishing in an AI-powered world will not be realized through another technology breakthrough. It will be realized through deliberate incremental extension of the norms and standards that have supported publisher integrity for centuries. As for the web, safe navigation in a corpus that welcomes large language models like ChatGPT is not impossible, but we may need to rethink how we manage online public information.
