The art of getting the correct words from AI
Tips and tricks to ensure your $4.6 million dollar language model is useful
Introduction
By now you’ve probably heard of ChatGPT, the large language model designed to answer questions and provide information on a wide range of topics. And if you’ve used it enough, you’ve probably come to dread these seven words
I’m sorry, as a large language model
As disappointing as it is that the world’s “smartest” AI refuses to provide step-by-step directions on how to rob a bank, what does that phrase even mean?
What is a Language Model?
Funnily enough, ChatGPT being a large language model has very little to do with why it responds the way it does. In it’s own words:
Q: What is a language model?
A language model is a type of statistical model that is designed to understand and generate human language. Language models can be thought of as “smart” computer programs that are able to predict the probability of a word or sequence of words occurring in a given context.
A Transformer-based language model is trained towards one and only one purpose: given a massive vocabulary of words, and a sequence of existing text, calculate the probability — for every single word in the vocabulary — that it’ll appear as the next word in the sequence.
For now, we can think of Transformers as a black-box doing just that. If you would like to read more about them, take a look at this amazing article https://jalammar.github.io/illustrated-transformer/
Due to this, language models have no understanding of the meaning or context of the text beyond what is provided in its input.
Due to this, language models are extremely keen on responding within the style and context of it’s input.
In order to generate text from a language model, we use a process called Autoregressive Sampling. This is a method for generating text using a language model that involves predicting one token at a time based on the previous tokens. In other words, given a sequence of input tokens, the language model predicts new token, which is then appended to the sequence. This process is repeated iteratively to generate a longer sequence.
Making Language Models give you the word you want
Sampling Techniques
Unfortunately, there’s a small problem with just taking the tokens with the highest probability(called greedy sampling): it often leads to very repetitive text, short responses, and even semantic problems. Notice the frustration in this crude attempt to make my very own ChatGPT:
With greedy sampling, the model is fully deterministic — the same input will always generate the same output.
Fortunately, there’s a better way to do this: sampling from the distribution of output probabilities.
We can go even further to increase the “randomness” of the model, by changing a property called temperature. For some background: neural networks don’t directly predict probabilities, instead they predict unscaled log probabilities, after which the Softmax function is applied.
All Softmax does is convert the log probabilities to probabilities using exp(), and then simply normalizing the probabilities so they add up to 1.
This allows us to do a neat trick, where dividing the log probabilities by a value, the temperature value, allows us to increase the probability of less likely tokens appearing.
>>> a = Tensor([1, 2, 3, 4])
>>> softmax(a)
tensor([0.0321, 0.0871, 0.2369, 0.6439])
>>> softmax(a / 2)
tensor([0.1015, 0.1674, 0.2760, 0.4551])
>>> softmax(a / 100000)
tensor([0.2500, 0.2500, 0.2500, 0.2500])
>>> softmax(a / 0.00001)
tensor([0., 0., 0., 1.])A lower temperature value makes the sampling more trusting of the model’s highest probability values. This does cause the rare mishap, however, but that can be taken care of by simple mitigations, such as only regarding the top K logits and/or the top P% highest logits.
An even fancier sampling trick is to consider multiple sequences at the same time, and selecting the one with the highest cumulative probability, in a process called beam search.
Prompting
If you’ve spent any significant amount of time trying to work with ChatGPT, this is probably a concept you’re familiar with.
Like we established before, language models have no sense of context except the one you provide it, which is why prompting is one of the most crucial aspects to getting reasonable outputs from a Large Language Model.
Just take a look at these two conversations on the OpenAI playground
This will probably make more sense when you see the text each conversation was prompted with:
Though this is a fairly exaggerated example, the point to take away here is that prompts are the only way to provide context to a language model, and the only way to get it to behave in a certain way. If you prompt the model with Interstellar dialogue, it will only try to continue the document as it was before, i.e. writing more Interstellar dialogue.
This is also often the reason why LLMs such as ChatGPT tend to respond so confidently with incorrect answers. It requires a fair bit of prompting effort in order to prevent these models from “hallucinating”.
In order to get the response we want from a language model, the best way to go about doing so is to set up an environment where the desired response would be likely to appear. There are also a few tips and tricks when it comes to prompt engineering in this OpenAI article.
An interesting emergent property, as a consequence of this context dependence, is that language models are sometimes very stubborn. Once there’s a response in the model input that contains, let’s say, ChatGPT writing an unfactual statement, there’s a much higher probability of it appearing again.
Just telling it the correct words
This one might seem a little obvious, but the best way to get the model to output what you want, is to restrict its output to those options.
Most people interacting with natural language models don’t do so in this way, but LMs can be very easily used for zero-shot classification. Simply plug the options you want to classify between into the model, and compare the probabilities it outputs for each of the options.
And this can be extended in a myriad of ways, a few of them are
- Command generation
- Code autocomplete
- Deciding what to search on the web
- Calculator prompts
In any use case that requires the generation of structured grammar, a language model can simply be plugged into an existing autocompletion system, like a Python language server, for example.
Reinforcement Learning
You probably weren’t expecting to see this here, but OpenAI has pioneered a way to incorporate (relatively) small scale manual labeling to manipulate a language model’s behaviour.
As illustrated in this infographic, OpenAI was able to train a reward model on human labeled responses, which they then used to fine tune GPT using Reinforcement Learning.
Conclusion
The model architecture which makes all of this possible is truly incredible. The Transformer has stood the test of time, going relatively unchanged since it was first introduced in 2017 in the paper “Attention is all you need”. Most of the improvements in AI since then have primarily focused on everything surrounding the Transformer — building massive corpuses of text data, coming up with efficient pipelines to train models with hundreds of billions of parameters, and of course, sampling the results form these models.
Sources
Holtzman, Ari, et al. “The curious case of neural text degeneration.”
Ouyang, Long, et al. “Training language models to follow instructions with human feedback.”
