ChatGPT Explained: Unveiling the Secrets of ChatGPT’s AI Technology
The field of Artificial Intelligence (AI) has witnessed remarkable advancements. Among the most intriguing innovations is ChatGPT, -the GPT stands for generative pre-trained transformer — a revolutionary language model that has redefined human-computer interactions.
The goal of this article is to be a simple explanation of the technologies that empower ChatGPT.
1- Foundation models and transformers
2- Finetuned LLMs (Large language models) with instruction finetuning
3- Finetuned LLMs with RLHF: Reinforcement Learning from Human Feedback
1/ Foundation models and transformers
Machine learning consists of training a model on data with labels and letting the model understand the pattern to identify the different classes. For example, if we want to teach a model to identify toxic comments. We give it toxic and nontoxic comments. Then the model is able to identify the words that will help detect the type of comment.
This is called supervised learning, on the other hand for language models the training is done with a self-supervised technique. The model is given unlabeled data and learns to perform the task by generating its own labels. This allows us to use big datasets for training.
Foundation models are very large neural networks trained with self-supervised learning on massive amounts of unlabeled data.
To put it simply, the model is trained to predict the next word in a sentence. This training allows it to develop a good representation of each word.
This representation is possible thanks to the transformer architecture, which uses attention. We will not go into more detail about this part, but to put it simply, attention is a technique that allows the model to understand the relationships between words in a sentence.
For a more technical explanation: http://jalammar.github.io/illustrated-transformer/
Now the model is able to understand any sentence. We only need to teach him the task at hand. It’s possible thanks to simple finetuning. This was the status of GPT models back in 2019. (GPT2)
How is it applied to chatGPT?
The starting point of ChatGPT creation is a pre-trained language model in the GPT-3.5 series.
2/ Finetuned LLMs with instruction finetuning
Thanks to instruction finetuning we are able to train the model on a range of different tasks and improve the zero-shot performance of language models on unseen tasks.
A great example of instruction finetuning is the Flan T5 model, an open-source LLM model from Google, Flan-T5 outperforms T5 at almost everything. For the same number of parameters, Flan-T5 models have been fine-tuned on more than 1000 additional tasks covering more languages.
Example of Instruction finetuning sentences used :
The model is now able to generalize to multiple tasks with no task-specific finetuning. Tasks like text summarization, and sentiment analysis … that needed the model to be trained on labeled data now can be easily accomplished by LLMs. This is called In-context Learning or zero-shot learning.
The implication is huge. Now automation is more accessible. Instead of needing a team to create a specific model to classify toxic comments, you can now ask the language model.
How is it applied to chatGPT?
In the ChatGPT paper, this is described as step 1. Human labelers answered to prompts, this new dialogue dataset was mixed with the InstructGPT dataset and the result created is used for a supervised training of the initial language model.
3/ Finetuned LLMs with RLHF: Reinforcement Learning from Human Feedback
This step is crucial in order to increase the communication skills of our model. Thanks to RLHF ChatGPT performs better in conversational applications.
To put it simply, we ask the models questions and then generate answers. We reward the model when the answers are similar to or better than human answers. This way we teach the model how to chat like a human.
To explain simply reinforcement learning let’s use a dog metaphor:
Imagine you have a dog that you want to train to sit. You could start by giving the dog a treat every time it sits. The dog will quickly learn that sitting is good behavior because it leads to a reward.
As the dog gets better at sitting, you can start to make the task more difficult. For example, you could ask the dog to sit in a different location or to sit for a longer period of time. The dog will continue to learn and improve its behavior as long as it is rewarded for the desired outcome.
Reinforcement learning is a similar process. In reinforcement learning, an agent is given a task and is rewarded for taking actions that lead to a desired outcome. The agent learns to perform the task by trial and error, and it gradually improves its behavior over time.
A foundational model has a good representation of the words and also the world thanks to the links between words he is able to make. Now thanks to RLHF the model is able to answer like a human which is the desired outcome.
More details below:
The start point of this step is the supervised fine-tuning model trained in the step before.
Then we create comparison data: In this phase, the labels are defined thanks to Human evaluators that are asked to rank the model responses based on their quality compared to the human references.
We then create a reward model that helps evaluate how good the model-generated responses are compared to human responses.
We finetune the pre-trained language model thanks to a reinforcement learning algorithm named Proximal Policy Optimization (PPO). This algorithm optimizes the model’s parameters to increase the likelihood of generating responses that receive higher rewards. The process is iterative, and the model is fine-tuned over multiple epochs, using batches of data from the comparison dataset.
How is it applied to chatGPT?
By using RLHF, ChatGPT learns from human feedback, allowing it to understand and generate more contextually relevant and coherent responses.
Summary of the explanation :
The starting point is a pre-trained language model. At this step, the model is able to have a good representation of the entities involved. Once the model has the knowledge, it needs to understand the best way to answer and return knowledge. We do this by showing the model how a human typically does it, using RLHF.
The information in this figure is from the paper of the InstructGPT model but as explained by OpenAI both models ChatGPT and InstructGPT use the same methods but with slight differences in the data collection setup.
Conclusion:
This article delves into the key technologies utilized in creating ChatGPT. However, it is essential to acknowledge its limitations, such as a tendency to produce inaccurate or biased information and its limited memory and context retention.
Nonetheless, I remain optimistic about the future of AI-driven conversations and stress the importance of responsible AI development for the betterment of humanity.
Sources :
https://jasonwei20.github.io/files/FLAN talk external.pdf
