Don’t feed meat to the model
Large Language Models are data-hungry beasts and at the same time they are able to generate huge amounts of data that is almost indistinguishable from human generated data.
So, what if we just… fed them their own output?
That’s like giving a cow a diet of beef — welcome to the AI version of BSE. Just like cows weren’t designed for a carnivorous feedback loop, LLMs start to lose the plot when they’re forced to binge on their own words. It’s a recipe for machine insanity — a digital echo chamber where each round of learning makes the output a bit weirder.
This is a very abstract problem, and it’s hard to get an intuitive understanding of what’s really happening “under the hood”. To stick with the BSE cow metaphor for a little longer– this is how the FDA explains BSE and what happened back then:
The parts of a cow that are not eaten by people are cooked, dried, and ground into a powder. The powder is then used for a variety of purposes, including as an ingredient in animal feed. A cow gets BSE by eating feed contaminated with parts that came from another cow that was sick with BSE.
There is a natural response that goes something like “we as a human race shouldn’t have done this” an intuitive feeling that this was bad and one could have known the upcoming disaster.
But that is the thing with hindsight — it’s not prescience. And the technical details about BSE are very, very complicated and unintuitive. I remember that I learned the word “prions” back then.
Mad computer disease
What does it actually look like when a Large Language Model (LLM) “goes mad”? I did read some articles about the math behind the problem. Lots of numbers and thresholds and problems, but no “aha” moments. I am writing this, because I recently witnessed my first AI spinning out of control, and it was more than a little unsettling. This time, I had my “oh this is bad …” moment. So, I’m here to share that experience.
To make sense of my story you will need a little technical context, but not much, I promise. Sometimes I open up ChatGPT and ask the model about an abstract technical problem we’re working on in our company, just so see if there is (on a high level) a way to approach a problem that we haven’t thought of.
The specific problem isn’t crucial in the context of MAD LLMs, but I’ll try to explain enough so that you can grasp the moment where the train did jump of the rails.
So what happened? I asked the LLM about a technical feature I knew didn’t exist. When language models begin to hallucinate, it’s often a sign that the model lacks sufficient information on the topic. It’s like being in high school, asked by your teacher about a book you should have read but didn’t — so you start making things up to fill the gaps. In the dialogue I was asking about AWS Route53 which is a DNS service. DNS is short for domain name service. It is the thing that helps your computer to find the right server to a domain name like medium.com. DNS is responsible for computers having names instead of numbers that are hard to remember.
Now, one domain name can point to a lot of servers. Your Netflix.com isn’t one machine. And there are different approaches on how to get the right endpoint to answer your request. The details are really not important here.
The complexities of DNS make it hard to manage more than 25 servers under a single (sub)-domain, and I was searching for an easier solution to this problem than the one we are currently using for delivering our data that is compatible with the DNS standard.
Now domains (for example www.test.com) can point to other domains (that is called CNAME) and it is something like an alias for a name. You can also call Alexander Alex — he will listen to both names. But unfortunately, it’s not possible to have one domain name point to a list of names.
So I already knew that this is not possible with AWS Route53 (the domain name service we are using) but I asked nevertheless, because I was hoping for an answer like “this is not possible with Route53, but you could try this approach” (e.g. I already knew that there is a proprietary way in AWS to do so, but I was looking for alternatives.)
So this is my question (I was typing on my mobile while waiting for a call to start and yes I know that this is not a question, but I knew the model would understand what I want):
User: cname round robin on amazon route 53
Answer: To set up a CNAME round-robin using Amazon Route 53, you can follow these steps:
What is CNAME Round Robin?
Round-robin DNS distributes traffic across multiple servers by resolving a single domain to different IP addresses (or hostnames) in a rotating manner. With CNAME round robin, instead of distributing traffic to different IPs, you use CNAMEs (Canonical Name records), which point to different domain names.
Steps to Set Up CNAME Round Robin on Route 53:
[…]
When you query your CNAME (e.g., www.example.com), you should see the different records being returned in a round-robin fashion.
[…]
This is a full hallucination. Except for the great definition of CNAME round-robin at the start (which shows that the model completely understood what I was searching for) the rest is complete BS — although it sounds exactly like the answer I was hoping for. Luckily I knew that and LLM hallucinations are kind of the price you pay for helpful models. So when I got this answer, I knew the language model was just answering my question in a plausible way and the technicalities of how the machine imagined CNAME round-robin to be implemented is just the way I was hoping existed. But still it’s a hallucination.
This isn’t necessarily a bad thing. When the model generates hallucinations, it’s often because you are moving beyond common knowledge and enter the realm of something new.
So with this answer on my screen and my call still not starting, I type: “are you sure — search” on my mobile. Which is my way of saying, please doubt yourself and use a web search to clarify the information provided. And this is the answer I got:
Amazon Route 53 does not support true CNAME round-robin directly. Instead, the typical round-robin approach uses A records or multivalue answer routing.** Here’s the breakdown:
[…]
2. CNAME with Route 53: You can technically create multiple CNAME records for the same domain in Route 53, but this isn’t typically how DNS round-robin is set up, as DNS generally prefers A records for round-robin purposes. If you still need to route across multiple CNAMEs, you can manually configure it, but note that round-robin functionality (randomized distribution of traffic) is not guaranteed as it depends on the DNS resolver behavior .
[…]
You can see that with the help of a web search the model was able to produce the right output at first (AWS does not support …) which is what I already knew. But then continues to provide the full wishful hallucinated and wrong answer in the second part.
This time with a citation hyperlink to a source.
So the model insisted on it’s hallucination, but it found a credible source. Someone believed in the same thing.
With my call still on hold I was interested, so I followed the link and there it was in plain sight: The madness. And the reason for this article.
This webpage from a company called Pulumi got all I was hoping for, and it described the solution to my problem in the simplest and clearest way. Except it doesn’t.
Pulumi — according to their homepage has one mission: To democratize the cloud for every engineer. I don’t know if that is a thing.
This website was generated by a LLM. It looks like they cache and beautify the answers of their chatbot. As they say on their page it’s generated with the Pulumi AI Project and to spice things up — nowhere in the meta-information of this page does it say that this page is AI generated.
This data is now going to train future models on how to solve problems like mine.
Thing is: This website looks like they know what they are doing. The “documentation” and the code snippet look exactly like what I am looking for. This is the cleanest possible way to solve my problem. If I find an equally looking website from a developer who describes how he solved a particular problem with code examples, I am willing to put a lot of trust into this person, because I know how much time and effort goes into writing good documentation and simple looking and understandable code. It is usually really hard to get an easy solution to a complicated problem and then to be able to show off in this “easy as one two three” manner. Not only in computer science, but in every field. Hard problems are … hard.
But this is just a honeypot of something that looks very much like helpful information. And instead of saving us time — this kind of information steals time from our society. It’s the shittification of our society.
Although I knew their code could not work I still doubted myself and setup a test environment to run their code. For the sake of completeness, this is the error they are running in. It is not possible to have more than one CNAME target on Amazon Route53.
But to get back to where I am coming from. I am not complaining that LLMs hallucinate. I do not think this is a problem by itself and perhaps it is even an early, simple form of creativity. If I am in a chat with a LLM and I ask about complicated things that I don’t know the answer to in a field I know something about — I don’t expect the LLM to know the answers, but perhaps to test me and my preconception in a way I haven’t thought about.
LLMs citing its own output as a reference to credibility is a totally different thing. This is someone claiming “This must be true, because I have said the same thing yesterday.” And of course no matter how good we are trying to presort and clean the training data, the next training run for the next generation of AI models is going to contain a huge chunk of LLM generated text. It is happening right now.
We have fed meat to cows.
So, how big of a problem is this?
For the time being it only makes the internet a place even worse than it has been. Social media is flooded with AI-generated content, driven by political or commercial agendas. We’re now entering a phase where technical documents contain hallucinations and are treated as credible sources. It is one thing if wishful thinking dominates the political debate and facts play less and less of a role, but we would really want the people who build bridges, banking systems and hospitals to have their technical information based on facts, knowledge and good practice.
Consider wordfreq — a software library that tracks word frequency — essentially, how popular certain words are over time in large datasets.
Wordfreq announced, that they will not provide new datasets because AI generated content spoiled everything. It doesn’t make any sense to analyze the popularity of words in our languages, when the data we rely on is no longer written by humans.
Meaning the amount of AI-generated data in our day to day texts and information is so huge, that we cannot get reliable output from human data — just because there is much more non-human data. Today.
The agency to control the food quality for cows stopped their work, because cow food is now mostly made of cows.
How do we move forward?
A lot of effort goes into flagging AI-generated content. Like adding a warning label for humans, but especially machines. But I doubt this will meaningfully solve the problem, because think of the motivation of the people who put AI-generated content on a webpage or in a book or in a technical documentation. They do this to present themselves as having an answer for everything — to be seen as a credible source for information, by humans, by search engines, by researchers, by businesses, by machines. The last thing they’d want to do is to put a warning label on their information that says “Do not trust us”. If you had to write a warning “Please do not read this book” on a book — you should not publish the book in the first place.
Perhaps our focus could shift a little from generating more and more information to getting rid of unhelpful or even harmful information but then again this also sounds like censorship and who is to decide which information is so misleading that it causes harm to us as a society?
The only way out that I can see at this moment are networks of trust. Humans have the capacity to trust one another. While not every human is trustworthy, trust itself is a hard currency — once earned, people are reluctant to loose it. I am yet uncertain on how the best way to build a network of trust for various data sources looks like in the age of AI, but I am certain of one thing: AI itself cannot play a role in it.