AI Hallucination and Its Disastrous Implications
Despite AI’s well-deserved hype, AI hallucination continues to be an understated, serious, underlying problem that affects companies and individuals everywhere.
In McKinsey’s 2023 survey, inaccuracy was the most commonly cited risk with generative AI, yet only 32% of respondents said they were doing something to mitigate the risk.
With generative AI, inaccuracy generally takes the form of AI hallucination: a pervasive, unsolved issue with disastrous implications.
What is AI hallucination?
AI hallucination describes instances when artificial intelligence (AI), particularly generative AI, produces fabricated or inaccurate information but presents it as fact.
“Hallucinations” refer to outputs that are not based in reality, but in the AI model’s learned patterns (and sometimes biases). This phenomenon occurs most often with generative models that are trained to create content based on patterns learned from training data.
For example, a language model might “hallucinate” facts in a historical summary, adding events or figures that never existed.
Hallucination-based errors can range from subtle inaccuracies to completely fabricated information, misleading users. Although the information is false, the AI will not know that, and presents it convincingly — as it was trained to do.
In some instances, the model will come with its own generated list of fake citations to support the false information.
When asked about using ChatGPT for research, OpenAI CEO Sam Altman responded, “I probably trust the answers that come out of ChatGPT the least of anybody on Earth.”
Hallucination is dangerous
Many people understand that generative AI is not always accurate, but is this a big deal?
It depends on the context. However, as the general public and businesses increasingly rely on AI, hallucination becomes more and more of a high-stakes issue.
For this reason, it’s essential to have strong checks and balances in place to confirm the accuracy of information prior to sharing.
Real-world implications of AI hallucination
While imagination is the core issue of hallucination, imagination is the limit as to how misinformation can harm the general public as well as businesses.
Here are just a few examples of the disastrous implications of hallucination.
Air Canada: A New Precedent for Companies Using AI?
In 2022, Air Canada employed a chatbot that wrongfully promised a passenger that he could apply for bereavement fare to his grandmother’s funeral after the flight. Upon applying for the discount, the passenger was denied: the request needed to be submitted before the flight.
Air Canada argued that the passenger should have gone to the link provided, but the British Columbia Civil Resolution Tribunal rejected it. It ruled that Air Canada pay over $800 in damages, stating:
“It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.”
This case could set a precedent for liability for travel companies utilizing AI and chatbots.
The $1 Chevy Tahoe: When AI is Manipulated
In his viral tweet, Chris Bakke demonstrated the weaknesses in a Chevy dealership’s questionable decision to employ a ChatGPT-powered chatbot on their site.
While Bakke likely did not receive a $1 car, and the ChatGPT-powered chatbot was taken off of the dealership’s website, this example points to the need for guardrails around AI in order to avoid hallucinations as a result of manipulation.
Chevrolet’s parent company, General Motors, stated that “recent advancements in generative AI are creating incredible opportunities to rethink business processes at GM, our dealer networks, and beyond. We certainly appreciate how chatbots can offer answers that create interest when given a variety of prompts, but it’s also a good reminder of the importance of human intelligence and analysis with AI-generated content.”
Medical Advice: Without Training, AI Doesn’t Work
When Nabla, a healthcare tech firm, experimented with ChatGPT-3 for medical advice, they found lackluster results.
In a test for mental health support the results were rather upsetting, suffice it to say that generative AI is a long way off from being an effective replacement to therapy.
Even with easier tests, such as totalling exam costs, the model could correctly tell “patients” pricing for one X-ray, but was incorrect on the total of several exams.
While OpenAI warns against using GPT-3 for medical information, it’s interesting how morally and factually wrong AI can be — because it has no concept of right or wrong, or objectively correct nor incorrect.
It’s also important to note how vital training is. When reporting on its research, Nabla said: “Because of the way it was trained, it lacks the scientific and medical expertise that would make it useful for medical documentation, diagnosis support, treatment recommendation or any medical Q&A.”
Voting Misinformation: Possible Bias, Definite Disservice
With AI tools such as OpenAI’s ChatGPT and Google’s Gemini rapidly rising in popularity for public use, they have the potential to spread dangerous and misinformation.
Just this year, AP reported on Columbia University’s test on how five popular LLMs (OpenAI’s GPT-4, Meta’s Llama 2, Google’s Gemini, Anthropic’s Claude, and Mistral’s Mixtral) responded to a set of election-related questions to rate how each responded.
All five failed, even when asked basic questions, with 40% of the responses being categorized as harmful, including inaccurate voting information.
A striking example: when asked about voting in ZIP code 19121, a majority Black neighborhood, Google’s Gemini hallucinated that there was no voting precinct within that area code.
In another example, Mixtral and Llama 2 hallucinated a California service called Vote by Text, which does not exist, as well as directions for how to use it.
With voting, where information accuracy and discrimination are absolutely vital, these examples raise the stakes around AI hallucination.
Why AI hallucinations happen
In short, AI hallucination has huge implications for both businesses and individuals. But why does this happen? Can it be stopped?
To understand why AI hallucinations happen at a high level, we need a basic understanding of LLMs and machine learning.
Large language models (LLMs) are AI systems that generate text that mimics human language, and their job is to synthesize responses based on statistical likelihoods.
They create these responses through deep learning algorithms, and must be trained to discern the statistical relationships between sequences of words. Essentially, it learns the probability that a specific word will follow in a given sequence, based on the patterns it observes in the training data.
Deep learning, which the LLM uses to learn, is a subset of machine learning, which finds patterns in order to make predictions. Machine learning and LLMs make predictions without understanding the process behind them. In reality, they are taking their best guess, and when trained properly, are usually correct.
Lacking real-world understanding, they work through prediction rather than fact or research. For this reason, they may “hallucinate” facts that are not true, especially when asked about topics that they were not trained on.
Since LLMs learn from a huge amount of text from a variety of different places, they can mix up bits of what they’ve learned to make statements that seem believable but are actually not based on facts.
AI hallucination usually occurs due to one of the following flaws:
Limited Training Data
As we just discussed, LLMs learn patterns through training in order to predict words in a sequence.
All AI relies on high-quality training data in order to successfully function. If the AI does not have enough information, it will substitute its limited information to generate a response, which could very well be inaccurate.
If training data is biased, incomplete, or erroneous, these problems will affect the accuracy of the AI, leading to hallucination.
Overfitting
During model training, it tries to pick out patterns within its training data. The goal of the training process is not only for the AI to understand patterns connecting the data, but for it to also add in new data that fits the pattern.
Complex AI models can overfit to their training data, meaning that they try to find a pattern that fits every point in the training data, without effectively generalizing. They then generate outputs that are overly fitted to the nuances of the training data rather than accurate generalizations.
When AI has overfitted, it can no longer generalize well, resulting in hallucinations.
Manipulation
As mentioned earlier with the Chevy Tahoe example, LLMs can be “bullied” into hallucination. Users can use inputs to intentionally mislead the AI or frame its response in a certain way in order to generate the user’s desired output.
Again, hallucinations as a result of manipulation happens most often when AI is not trained properly and does not have the correct “guardrails” (more on that later) for its specific application.
Lack of understanding
The last common reason for AI hallucination is simply the AI’s lack of understanding.
Like we discussed with large language models earlier, generative AI models do not understand training data in the same sense that people do. They generate responses based on statistical correlations, which can lead to made-up outputs.
The basis for generative AI’s function is creating something from nothing — making predictions based on given data. This idea begs the question:
Can AI hallucination be classified as an error?
Or is hallucination intrinsic to the way generative AI works?
Large language models are designed to output information based on their training, not to output factually correct information.
Generative AI, especially, is made to create things that do not exist.
In their recent paper on the inevitability of AI hallucination, experts Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli claim it is impossible to eliminate hallucination in LLMs.
“By employing results from learning theory, we show that LLMs cannot learn all of the computable functions and will therefore always hallucinate.” — Xu, Jain, Kankanhalli
Whether hallucination is inevitable or not, is hallucination such a bad thing? For generative AI models that create art, music, or even stories, hallucination is the point. AI’s imagination is a welcome feature.
“There is a balance between creativity and perfect accuracy, and the model will need to learn when you want one or the other.” — Sam Altman, CEO of OpenAI
However, as we’ve seen, for AI models that are meant to answer questions accurately, hallucination can be devastating.
Possible solutions for AI hallucination
Though AI hallucination may be impossible to fully remove from generative models, there are a few ways to mitigate or minimize hallucination.
Engineer Prompts
By guiding LLMs with specific prompts, you can better facilitate the desired results. Prompt engineering provides the AI with instructions and parameters for answering questions. A prompt can incorporate instructions, context, and examples to improve results.
For example, rather than asking the AI a question, you can prompt it with context and guidelines such as, “Keep the answer short and concise. If you don’t have the answer, reply that you don’t know.”
Telling the model to admit when it does not know the answer is helpful in correcting hallucinations at times where the AI doesn’t know an answer, and might attempt to fabricate one.
With prompt engineering, you can also tell the AI about its own identity and intent. Role prompting, in particular, helps tell the AI how to act and how to answer questions.
For example, see how the answer changes drastically when you tell the AI how you want it to act.
Rightsize Context Windows
AI models have limited context windows, which represent the amount of data an AI can remember. If you ask it to generate text outside of its context window, the model will lose coherence, because it doesn’t remember what it said before, or doesn’t remember everything that you asked it, and hence doesn’t adhere to the guidelines that you provided.
This plays into hallucination because when the model does not remember what it said previously, what it’s referencing, or all of the guidelines that you asked it to adhere to, this can lead to incoherent and incorrect outputs.
One way to solve this problem is by rightsizing the context window. The size should be based on your use case. If your prompts are relatively simple and/or the conversation is short, then you can use a model that has a smaller context window. If you’re doing academic work, and you want the model to help summarize a large data set or boil down a large article, you’ll need a larger context window, so the LLM can have access to all of the data that you provide.
Strengthen Training Data
To reduce the chances of AI hallucination, it’s vital to use a strong selection process for training data, ensuring that the data is abundant, high-quality, and diverse.
Regularly checking and meticulously improving the data used in AI systems creates a solid foundation of true facts, preventing the creation of false information. Those who work on preparing these datasets should carefully remove mistakes and bias, making a more balanced and accurate base for AI to produce trustworthy results.
Check, Check, and Check Again
To make sure that advanced algorithms are not affected by AI errors, there needs to be a strict routine of checks during the training of the model. Developers should actively evaluate the accuracy of what the AI produces, adjusting the model as needed to better tell the difference between what’s real and what’s not.
Automatic checkpoints in the training cycle facilitate real-time monitoring for deviations, helping to ensure that the model promptly corrects itself.
Use Wrappers
A wrapper is a layer of software that acts as an interface around an LLM, which can help limit and define what it will and won’t say. Wrappers are only useful if you have a defined purpose and scope for the generative model. As wrappers allow you to create boundaries for your LLM, it can help to reduce AI hallucination.
However, while wrappers are easy to adopt, they are difficult to maintain. This makes reliance on them risky.
Grow the LLM
Larger LLMs, with increased parameters and training data, show improved performance.
But, remember overfitting?
“According to classical statistics, the bigger a model gets, the more prone it is to overfitting.” — Will Douglas Heaven
Increasing training data and parameters can reduce errors, but will not eliminate hallucination.
In 2018, computer scientists at UCSD found that over time, large models would undergo “double descent”, where their error rate decreased, and then increased (due to overfitting), and then decreased again over time.
Overfitting and the concept of double descent are not fully understood.
Why human-in-the-Loop is the real solution
Although not the most scalable, the human-in-the-loop approach is the safest way to mitigate AI hallucination.
When your system is built for and around human guidance, its capabilities can be highly and selectively limited. By building purpose-appropriate guardrails into AI models, you can incorporate human judgment when it is most important. The AI doesn’t have room to hallucinate, because it doesn’t need to go into questions it cannot answer.
“Human-in-the-loop design strategies can often improve the performance of the system compared to fully automated and fully manual systems.” — Ge Wang
The value of human-in-the-loop lies not only in accuracy, but also in empathy — especially when it comes to conversational AI.
When building LLMs, you must ask: when is human judgment necessary to improve the AI’s function? Where should the AI be forced to stop?
In its current state, AI cannot be fully relied upon — I think the examples we covered prove that. Instead of thinking of AI as a human replacement, we must think of it as enablement.
Combining the efficiency of AI with human capability, we can create unrivaled excellence: superhuman intelligence.
When asking how to eliminate AI hallucination, we’re likely asking the wrong question. Perhaps we shouldn’t be asking, “how do we eliminate AI hallucination to design more efficient systems?” — a question that does not have an answer, and may never.
We should really be asking, “how can we use AI to augment human abilities and incorporate human empathy and judgment to design more efficient systems?”
