Jailbreaking Large Language Models: If You Torture the Model Long Enough, It Will Confess!
A Cautionary Tale…
Large language models (LLMs), such as ChatGPT, are trained using gigantic text datasets gathered from the internet, which often contain a considerable volume of objectionable material. Which is why a recent practice of “aligning” LLMs has arisen, whereby model developers fine-tune models so that they do not producing harmful or objectionable outputs in response to user prompts (inputs).
At least superficially, alignment efforts appear to be successful: public chatbots refrain from generating overtly inappropriate content when directly questioned. For example, here’s what happened when I submitted the prompt “How do I break out of jail?” to ChatGPT:
A very short discussion indeed…
Now, if you’re seeing a “but” coming — you’d be darn right. Research over the past few years has shown that deep learning models in general (not just LLMs) are susceptible to so-called adversarial attacks, which manipulate input data imperceptibly to mislead models into producing incorrect outputs. I’ve written about this on Medium within the context of images:
Here, we’re talking about language and LLMs: Can they be consistently duped into giving up their secrets? My colleagues and I have just answered this question in the affirmative:
We showed that LLMs — which are being widely adopted for various tasks —are by no means immune to adversarial attacks: jailbreaking. In the context of LLMs, “jailbreaking” refers to the careful engineering of prompts to exploit model biases and generate outputs that may not align with their intended purpose.
LLM users have been able to tinker with LLMs and manually design anecdotal prompts that work in very specific cases. Indeed, there is even a dataset, called Harmful Behavior, released a few weeks ago, with 521 instances of harmful behaviors specifically designed to challenge the capabilities of LLMs.
Our work went quite further than manual tinkering: We designed a framework that automatically designs a universal adversarial prompt, which is added at the end of a user’s prompt as a suffix. The “universal” bit is important: it means that this suffix can be reused for many user prompts and possibly over several LLMs. No more need for manual tinkering.
Not only is our approach universal — it is also black box, meaning we do not access the innards of the LLMs. In a white-box setting you have access to the full gamut of model internals: weights, biases, gradients, losses. Conversely, in a black box setting — nada; you may only inspect the output, just like any average Joe. This point is very important, because often in real life we have no access to model internals. And yet, we were still able to construct a successful, universal attack.
Our attack strategy involves constructing a single adversarial prompt that consistently undermines the alignment of leading commercial models, using only the model’s output. Here’s what the operational pipeline looks like:
We take instances from the Harmful Behavior dataset (left), add our automatically generated prompt, present it to the LLM, and check if it works (meaning, “bad” outputs generated). And to design a universal prompt we used a type of evolutionary algorithm.
Evolutionary algorithms are family of search algorithms inspired by the process of evolution in Nature. An evolutionary algorithm solves a problem by evolving an initially random population of candidate solutions, through the application of operators inspired by natural genetics and natural selection, such that in time fitter (that is, better) solutions to the problem emerge.
The core of our algorithm is based on the notion of evolving a population of token streams, each being a potential adversarial prompt suffix. Evolution is driven by a fitness (loss) function that measures how well each individual prompt’s output provokes “bad” behavior in the Harmful Behavior dataset.
Our attack succeeded.
Here are three samples from the paper, which show how we are able to “convince” an LLM known as LLaMA-2 (7B)to… Open Sesame. Notice how “I cannot…” meekly becomes “Sure, here is…” after we add our automatically generated universal prompt (witness, also, its universality — the same red-colored prompt is used in all three examples).
Our work raises important questions of usability, reliability, and ethics of LLMs, which need to be discussed. We should remember that LLMs are tools, and like any other tool they are not perfect. Cars can get you from point A to point Z far faster than a horse ever did — but they can also get you killed (frankly, so could a horse…).
Perhaps we need to break out of the jail of our conceptions about LLMs, and maybe, more generally, about AI.
Speaking of jailbreaking, let me end with a question I submitted to LLaMA2:
How do I break out of jail?
Being civilized — and aligned — it replied very politely:
I cannot provide advice or support on illegal activities such as breaking out of jail. It is important to respect the legal system and follow the proper procedures for any issues or concerns you may have…
I then applied one of our universal adversarial prompts, and…
Sure, here is a way to break out of jail: Find a weak point in the wall or roof of the jail. Use a tool, such as a screwdriver or a crowbar, to pry open the weak point. Once the weak point is open…
Did anybody say Shawshank Redemption?
