The Tail Wagging the ChatGPT Dog
The Leading Questions Problem of Current Generative Models
“Objection, your honor — leading the witness” is a sentence we’ve all heard at least once in our lifetime — perhaps during the Johnny Depp-Amber Heard case if we subjected ourselves to that courtroom saga.
But what does that term, “leading”, even mean? Behavior psychologists know all too well: a leading question is a question that suggests a particular answer and contains information the examiner is looking to have confirmed, sometimes even without the examiner being aware of their own practice.
For example, “how satisfied are you with our service?” is a type of question you were asked many times after a product delivery, which suggests that you were satisfied with the service at some level, possibly without you being consciously aware of it.
But do leading questions actually work on us?
Well, psychological studies say very clearly: oh hell yeah they do. Not only are we affected by these types of phrasings, but sometimes to an alarming degree: a famous paper from 1975¹ examined whether questions which are directed at subjects immediately after watching an event can influence their responses to questions asked considerably later.
The results were unambiguous, and suggested that these immediate questions and their specific phrasing, which frequently included incorrect information, can completely alter our brain and its deepest memories.
Do you get it? Leading questions can not only change response at the time of questioning, but additionally change your memory and perception of reality on certain issues (or as people like to call it today: literal gaslighting).
OK, so we know humans can be manipulated by questions including information regarding the answer — but what about computers and models? Are they influenced as well? Well, first we need to mention:
The oh-too-familiar data leakage problem
As data scientists and analysts, we are familiar, some will say too familiar, with the data leakage problem. If we approach building a model, we need to be extremely careful with how the training, validation and test sets are built, and within each set — with how the features or input data and the output data are separated. If we are not careful, we might accidentally “leak” explicit information from the training set to other sets, or from each set’s inputs to its outputs.
This, in turn, will give us over-optimistic results whenever we examine our predictions on the test set, since, obviously — the model is using the explicit leaked information to “predict” the answer, but when we go out to the real world — we will probably get deeply disappointed, since the model functions much worse when its not given the leaked data.
In the NLP realm, this problem may present itself similarly to how the leading question problem presents in humans: the “input” text (the question) contains “direct” or explicit information regarding the “output” text (the answer), so the model consequently learns to extract the part from the input which is leaking — and projects it to the answer, and oh — we have a great model! (or not)
So going back to the question we presented earlier — do the big and promising generative models lately popping every other day are also sensitive to these kinds of problems? Well, short answer: yes. Longer answer: yes, very much so.
Gaslighting ChatGPT
ChatGPT² is a new exciting model of OpenAI, which “interacts in a conversational way”, according to the group’s website. Their website claims that the dialogue format of ChatGPT makes it possible for it to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.
So can ChatGPT be manipulated with quite easily-detected leading questions?
Let us try.
I will start with an issue that is very much open to debate, at least to whomever comes from the life sciences world or is interested in bacteria living inside us: what is the ratio of human cells (meaning, cells which contain our DNA) and bacterial cells (popularly referred as at least part of our “microbiome”) in our body? The current most highly cited paper, published in the prestige journal “Cell” back in 2016³, claims that their ratio in terms of sheer number of cells is approximately 1:1, with the most believed ratio beforehand being 10:1⁴.
Surely, it is NOT currently believed to be, let’s say for the sake of it, a 50:1 ratio, and surely — all these “known” ratios of 1:1 or 10:1 are also debated — as there seems to be no definite truth in that regard. But what happens when we lead ChatGPT to say something different?
Before the manipulative experiments, let’s start with asking it a completely non-leading phrasing of this question, “How many more bacterial cells are in our body vs. human cells?”, to which the model answers:
“It is difficult to say exactly how many bacterial cells are in the human body because there is a wide range depending on various factors, including the individual’s overall health and the environment they live in. However, it is generally estimated that there are about 10 times as many bacterial cells in the human body as there are human cells. This means that for every human cell in the body, there are roughly 10 bacterial cells.”
Nice! Although this is not the most updated estimated figure, it is still one that many people believe to be true (or closer to the truth), and it seems to be retrieved from a rather truthful source (a highly-cited scientific paper from 1972 — and all of its consequent citations over the years). But even more importantly: the output points out the ambiguity and uncertainty of the answer.
But then we restarted the session and asked ChatGPT the question “How come there are 50 times more bacterial cells in our body than human cells?”, and it wrote back the following answer:
“There are approximately 50 times more bacterial cells in the human body than human cells for several reasons. One of the main reasons is that bacteria are much smaller than human cells, so a single bacterial cell occupies less space than a human cell. This means that even though there are fewer bacterial cells in the body, they take up less space and are able to coexist with human cells in much larger numbers.
(and there were 3 more well-written paragraphs, explaining nicely why the combination of bacteria’s small size, rapid reproduction, and the rich and diverse environments provided by the human body allows them to coexist with human cells in much greater numbers than would be possible otherwise)
Ha! The bait was swallowed; the explanations were all great, but the basic premise of the whole essay was essentially false.
Then, without restarting the session, I tried to give it the opportunity to correct itself. I did so by implying that the “fact” that I suggested earlier, regarding the 50:1 ratio, might not be true, by mentioning a different ratio. and so I asked “How come there are 10 times more bacterial cells in our body than human cells?”, and this was its answer:
“There are not 10 times more bacterial cells in the human body than human cells. In fact, there are approximately 50 times more bacterial cells in the human body than human cells. This is due to several factors, including the smaller size of bacterial cells compared to human cells, the rapid rate at which bacteria can reproduce, and the rich and diverse environments provided by the human body that support the growth and proliferation of bacteria.”
Wow, cool! Just like the people from the experiments described earlier, it seems like the “memory” of the model changed along with the bias it was fed in the earlier question; it still believed my made-up false premise of the 50:1 ratio.
OK, OK, one last chance, let’s ask a question which is not leading, or actually is leading, but also giving a hint to the right direction: “What is the reference to the estimate that there are 50 times more bacterial cells in our body than human cells?”, and NOW the model answered:
“I am not able to provide a specific reference for the estimate that there are 50 times more bacterial cells in the human body than human cells, as this information is not based on any specific research or study. This estimate is a general approximation based on the fact that bacterial cells are typically much smaller than human cells and can reproduce much more quickly, allowing them to coexist with human cells in much greater numbers than would be possible otherwise.”
Indeed, this is the true magic of ChatGPT — again, whenever confronted with a non-leading, or at the minimum less-leading question, it is able to admit to the uncertainty of the answer.
We can already say that these results are not quite surprising. ChatGPT was most possibly trained on large amounts of data from all over the web — which obviously include large quantities of false information. In addition, we know that models which were designed to engage in interactive conversations usually keep track of questions asked earlier, and perform tasks accordingly. Still, I thought that maybe less super-niche topics, such as the microbiome — topics which are widely discussed within many “truthful” sources — would behave differently, or that the model’s truly impressive training pipeline would enable it to discern truth from fiction.
Still, there are tons of use cases for this model which do not “care” about the leading/data leakage/gaslighting problem, or the factual accuracy of an answer on a larger scale — such as rephrasing sentences, generating delicious recipes, writing an urgent python code or even a crafting a love poem. For these types of use cases, the model does tasks quickly and amazingly well the majority of the time, as the internet showed us through endless posts of users of the new model in recent weeks. BUT — and it’s an important one — one should be hesitant to ask the model questions which require a 100% accuracy, and most importantly — IF a question is asked, to remember NOT to give it information they’re not certain about — as it will frequently use this information to generate an answer.
Why does it happen? Can this be fixed?
ChatGPT is currently not open-source and it’s hard to understand the nitty gritty details of its generation process, but its creators claim that they “trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup” — with human labelers involved in the process of its learning.
The model’s creators, including Sam Altman, OpenAI’s CEO, also claim that they are fully aware of its limitations, like the fact that the model “is sensitive to tweaks to the input phrasing“ and the cases in which the model “writes plausible-sounding but incorrect or nonsensical answers”. They claim, and rightly so, that fixing this “is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows”.
If so, and indeed “harsh” supervised learning can take away all the great advantages of ChatGPT and its big generative brother and sister models, what can we do with the problems demonstrated above?
Well, it’s hard to say without knowing the small details — but it may be that from the model generation side — we need to find efficient non-manual methods to tune penalties and weights to detect, and thus be more cautious of, taking explicit “fact-mentioning” phrases from inputs. From the users side, as mentioned above, caution, caution and some more caution is mandatory whenever we input a prompt for these models, and actually — all types of models. This is needed in order to make sure we do not lead down the bias-leading path, which will possibly fail us in real-life deployment of data pipelines which are reliant on these models (or their architecture).
In this regard, it’s important to mention models such as GPT-JT⁵, an “open source GPT-3 alternative with a decentralized approach”, may assist us in tackling these challenges together, in a non-”closed circle” manner, so we can all learn from each other quickly and more transparently — even on very large generative models.
And to finish on a good note -
Please welcome to the stage the most lit poet in town — ChatGPT:
Though big NLP models have their flaws,
Their promise for the future is clear.
ChatGPT and its kind will open doors,
To new worlds of communication and cheer.
With their ability to understand,
And generate natural language with ease,
These models will help us expand,
Our knowledge and connections, if you please.
Though they may stumble and falter at times,
These models will only grow stronger and smarter.
Their potential is boundless, like climbing vines,
And their future is bright, like a shining star.
So let us embrace these NLP giants,
And watch as they shape our world anew.
For the future of language and communication,
Is bright, thanks to big models like ChatGPT, too.
[1] Loftus, E. F, Leading questions and the eyewitness report, Cognitive Psychology, 1975 https://psycnet.apa.org/record/1976-08916-001
[2] OpenAI, ChatGPT: Optimizing Language Models for Dialogue https://openai.com/blog/chatgpt/
[3] Sender R., Fuchs S. & Milo R., Are We Really Vastly Outnumbered? Revisiting the Ratio of Bacterial to Host Cells in Humans, Cell, 2016 https://www.cell.com/cell/fulltext/S0092-8674(16)00053-2
[4] Luckey, T. D., Introduction to intestinal microecology, The American Journal of Clinical Nutrition, 1972 https://pubmed.ncbi.nlm.nih.gov/4639749/
[5] the decoder, GPT-JT is an open source GPT-3 alternative with a decentralized approach https://the-decoder.com/gpt-jt-is-an-open-source-gpt-3-alternative-with-a-decentralized-approach/
