Comprehensive Analysis of LIMA: (Less is More for Alignment)
Until now, one of the main components of the GPT4, GPT-3.5, and ChatGPT models training has been the idea of RLHF (Reinforcement learning with human feedback).
Recently Meta published a research paper on its own new language model called LIMA. (Source: LIMA: Less Is More for Alignment (arxiv.org)
This study is primarily debunking the myth of RLHF by demonstrating that, given a really good dataset, it is possible to train a supervised model that can perform almost as same as GPT-3 or in fact better than Google’s BARD and in some cases like GPT-4 equivalent.
What is LIMA?
LIMA model is based on Meta’s LLaMA model, which has 65 billion parameters. LIMA stands for ‘Less is More for Alignment.’ It is designed to match the performance of GPT-4 or Google’s Bard.
The essence of the research paper on LIMA revolves around the principle of “Less is More,” and what matters the most is the Quality, Quantity, and Diversity of the training data.
This model is fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modelling.
In a carefully conducted human preference evaluation, it was found that in 74% of cases, the responses generated by LIMA were either equivalent to or significantly preferred over those produced by Alpaca-65B parameters model. Similarly, LIMA responses are either equivalent or preferred to the DaVinci003 model in 65% of cases.
Let’s first explore the training process of LLaMA before delving into LIMA.
LLaMA background:
Meta AI (Artificial Intelligence) has developed LLaMA, which is a set of foundational language models with parameters ranging from 7 billion to 65 billion. It stands for ‘Large Language Model Meta AI’ and it is designed to help researchers advance their work in this subfield of AI.
You can find additional details about LLaMA by following this link: Exploring the Capabilities of LLaMA
Similar to other language models, LLaMA operates by receiving a series of words as input and uses this information to predict the following word, thus generating text in a recursive manner.
LLaMA is made available under a non-commercial licence with an emphasis on research use cases to preserve its integrity and guard against exploitation.
Academic researchers, people connected to organisations in government, civic society, and academia, as well as international corporate research laboratories, will all be able to use the model on a case-by-case basis.
To know more about LLaMA check out this: LLaMA: Open and Efficient Foundation Language Models, by Meta AI
LIMA and LLaMA differ in what ways?
- The difference is that Meta did only fine-tuning with 1000 carefully curated prompts and responses, instead of very extensive training with lots of human feedback (RLHF) like OpenAI.
- The research paper tells RLHF, or human feedback is not the key ingredient what matters the most is the Quality, Quantity, and Diversity of the training data.
- Researchers have taken extra care in curating these one thousand prompts & responses where responses (outputs) are stylistically aligned with each other, but the prompts (inputs) are diverse in style.
Meta refers to this discovery as the “Superficial Alignment Hypothesis.”
It suggests that the alignment phase following pre-training mainly focuses on teaching the model a specific style or format that it can remember and use when engaging with users.Thus, fine-tuning is more about style than substance.
What is Superficial Alignment Hypothesis: (Important part of the research paper)
The hypothesis from the research paper suggests that:
“A model’s knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which sub-distribution of formats should be used when interacting with users.”
This simply means in the entire thing that the model learns is part of the pre-training which is the first layer.
- Pretraining of LIMA has been done by using the 65 billion parameters which was used in LLaMA model.
The Alignment phase comes after the pretraining stage in the second layer. This phase involves instructing the model on how to select a specific sub-distribution format among pre-trained distribution when interacting with the users.
- For this alignment phase, they have used 1000 carefully curated prompts and responses.
If the hypothesis is correct, and alignment is largely about learning style then the corollary of Superficial Alignment Hypothesis says:
“One could sufficiently tune a pre-trained language model with a rather small set of examples”
However, it is important to remember that prior to the alignment phase, which is the pretraining phase will still need a huge amount of data.
Dataset used in training LIMA:
For their experiment, Meta’s researchers gathered information from three diverse sources: Stack Exchange, wikiHow, and the Pushshift Reddit Dataset.
Additionally, the researchers themselves wrote 200 training prompts and provided high-quality answers that were manually authorized for LIMA’s training.
However, the research paper does not specify the exact datasets used for training, & these datasets are publicly not accessible. And the source code of LIMA is not yet available.
LIMA training:
LIMA is trained using the following protocol: -
- First, they start with LLaMA 65 billion parameters model for the pretraining phase.
- Then fine-tuned the 1k alignment dataset.
- To distinguish between the user and the assistant, a special end-of-turn token (EOT) is introduced at the end of each statement. It serves the same purpose as the end-of-sentence (EOS) in stopping the generation.
Note: The research paper does not reference the ‘End of turn token’ (EOT) character.”
LIMA Performance evaluation:
LIMA was evaluated by being compared to other models, including Alpaca, Davinci003, BARD, Claude, and GPT-4. Crowd workers performed the comparison first, followed by GPT-4. The crowd-workers compared the answers provided by LIMA with the responses from other models and determined which one was the most accurate or preferred.
Figure 1 shows the results of the human preference study, while Figure 2 displays the results of GPT-4 preferences.
- In the human preference evaluation, it is shown that responses from LIMA responses are either equivalent or preferred to the Alpaca-65B parameters model in 74% of cases.
- Similarly, LIMA responses are either equivalent or preferred to the DaVinci003 model in 65% of cases.
- What is striking about this result is the fact that DaVinci003 was trained with RLHF, a supposedly superior alignment method.
- But BARD provides better responses than LIMA 42% of the time.
- Finally, we see that Claude and GPT-4 generally perform better than LIMA.
Figure 2 shows the results of preference evaluation using GPT-4. Ironically here it is shown that even GPT-4 prefers LIMA outputs over its own 19% of the time.
Limitations
- The mental work required to create such examples is significant and challenging to scale up.
- LIMA is not as robust as product-grade models.
- A weak response generated is frequently caused by a bad sample during decoding.
Conclusion
The research paper on LIMA (Less is More for Alignment) highlights some key findings and insights. Here are the main takeaways:
- Quality, quantity, and diversity of the training data play a crucial role in training a model.
- The “superficial alignment hypothesis” presented in the paper suggests that the alignment phase after pretraining primarily focuses on teaching the model a specific style or format of interaction with users, rather than extensive fine-tuning processes like RLHF.
- The study highlights that the use of RLHF is not always necessary, showcasing that the LIMA model can achieve effective performance without relying heavily on it.
- This approach offers advantages in terms of manpower and cost, as it reduces the need for extensive human feedback and manual fine-tuning processes.
- It is important to note that even though the LIMA model is not open source and lacks a commercial license, the purpose of this research is to demonstrate different methodologies and their potential.
Overall, LIMA demonstrates that with a carefully curated dataset and appropriate training techniques, it is possible to achieve performance levels comparable to advanced language models.
About the Author:
Vaishnavi R is a Junior Data Scientist here at the Version 1 Innovation Labs.
