Harnessing the Power of Synthetic Data in the Era of Large Language Models (LLMs) and Generative AI
As we transitioned from the traditional AI methods of the 2010s to deep learning-based approaches in the 2020s, there’s been a marked shift towards leveraging large-scale datasets. These datasets are so vast that relying solely on human annotation is impractical.
In the era of traditional AI, the emphasis was on hand-engineering features. Machine learning models were trained on relatively small, high-quality, human-generated datasets. In contrast, deep learning architectures have pivoted away from manual feature engineering. Instead, they favor end-to-end models that, given a sufficiently large dataset, can discern features autonomously.
The rise of large language models (LLMs) as foundational tools has ushered in innovative methods to generate synthetic data. These models can then utilize this data to self-improve without human intervention. This capability is a stride towards achieving Artificial General Intelligence (AGI) — systems that can self-correct and learn continuously.
However, not all LLMs are created equal. Many capabilities exhibited by LLMs are emergent behaviors. For instance, reasoning, a vital trait for AI agents, is believed to manifest in models with over 100 billion parameters. Given the constraints of consumer hardware, running models beyond 7B to 13B parameters is challenging, especially without a substantial GPU budget. However, as of September 2023, there are exciting advancements in parameter-efficient techniques and the Mixture of Experts (MoE) approach.
Interestingly, based on the scaling laws of LLMs, there’s speculation that we might soon exhaust the available text data for further enhancements. Models like PaLM and GPT-4 have been trained on trillions of tokens. This has led to intriguing legal debates surrounding the data used for pretraining, especially when sourced from platforms like Reddit, Twitter, Facebook, and behind paywalls. Users need to exercise caution, as leveraging models trained on proprietary data could lead to legal complications.
Setting pretraining aside, this post delves into the task-specific adaptation of LLMs and the pivotal role of synthetic data.
One might ask, “Why the need for synthetic data? Aren’t LLMs like GPT-4 and PaLM already versatile solvers?” Indeed, these models can achieve state-of-the-art results, often surpassing human accuracy. However, there’s always room for improvement and specialization.
Here are some key papers that emphasize the generation of synthetic data, either for self-enhancement or for training more compact models. This process, frequently termed as “knowledge distillation” or the “teacher-student” method, involves a larger model (the teacher) producing labeled data for a smaller counterpart (the student).
Self Align
— **Title:** Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
— **Idea:** The approach uses principle-driven alignment in addition to prompts, similar to constitutional AI. However, the authors distinguish this method as “alignment from scratch,” meaning it doesn’t rely on a RLHF LLM for bootstrapping.
— **Link:** [Read more](https://arxiv.org/abs/2305.03047)
- Self Instruct
— **Title:** Aligning Language Model with self Generated Instructions
— **Idea:** The core concept is to bootstrap instruction data from an instruction fine-tuned model using in-context learning. Subsequent fine-tuning of the LLM on additional instructions allows it to outperform larger models that use private datasets.
— **Link:** [Read more](https://arxiv.org/abs/2212.10560)
- Self Consistency
— **Title:** Self Consistency Improves Chain of Thought Reasoning in LMs
— **Idea:** The method employs CoT and various prompts to generate answer puzzles. The majority vote determines the label, and sampling is preferred over greedy decoding.
— **Link:** [Read more](https://arxiv.org/abs/2203.11171)
- Self Refine
— **Title:** SELF-REFINE: ITERATIVE REFINEMENT WITH SELF-FEEDBACK
— **Idea:** An LLM generates a draft, and another LLM refines the document, receiving feedback. This iterative process enhances the base LLM’s performance on the task.
— **Link:** [Read more](https://arxiv.org/abs/2303.17651)
- Self Debug
— **Title:** Teaching Large Language Models to Self-Debug
— **Idea:** The LLM learns to analyze its own generated code through execution and introspection.
— **Link:** [Read more](https://arxiv.org/abs/2304.05128)
- Self-Eval
— **Title:** Language Models (Mostly) Know What They Know
— **Idea:** The LLM is used to self-evaluate its output for correctness in various settings. The focus is on understanding P(IK) — how well the model calibrates its confidence.
— **Link:** [Read more](https://arxiv.org/abs/2208.08094)
- Generative Agents
— **Title:** Generative Agents: Interactive Simulacra of Human Behavior
— **Idea:** The paper describes an agent architecture that uses a large language model to generate believable behavior. It comprises three components: memory stream, reflection, and planning.
— **Link:** [Read more](https://arxiv.org/abs/2304.03442)
- Camel
— **Title:** CAMEL: Communicative Agents for “Mind” Exploration of Large Scale Language Model Society
— **Idea:** Two agents engage in problem-solving, exemplified by an instructor and programmer. They collaborate in a conversation to complete a task.
— **Link:** [Read more](https://arxiv.org/abs/2303.17760)
These papers represent just a fraction of the ongoing research. Moreover, these strategies aren’t confined to text-based models. Multimodal LLMs, which combine text with other modalities like vision or speech, can also benefit from synthetic data. The challenge with multimodal LLMs lies in aligning different modalities. For instance, an image might contain multiple focal points that need to correspond with relevant textual descriptions.
