Bored over the holiday season? Try prompt injecting a customer service chatbot
Chris Bakke has posted a screenshot on X showing how, through a very simple prompt injection, he got a Chevrolet customer service chatbot to sell him a late-model Chevrolet Tahoe for a dollar, and say it was a legally binding offer.
One assumes Chevrolet will not follow through on the deal… The chatbot in question was a simple adaptation of ChatGPT 3, and shows what can happen when companies decide to replace their customer service teams with a rudimentary chatbot.
Bakke has raised an important question: if you design a chatbot based on a generative algorithm to accept any instruction typed by a customer, then it will not just process the information and respond, but will also interpret it as conditioning factors for future conversations in that thread. In his experiments with the Chevrolet chatbot, Bakke also managed to get the company to offer him a two-for-one on all its vehicles and even sell him a Tesla.
Prompt injection is when an end user of an LLM application (or any generative AI application) gives it instructions to make it bypass those the developer of the application have provided. By means of prompt injection, all kinds of responses have been obtained, including for example details about the training of certain algorithms.
