[Research Paper Summary]ReFT: Representation Finetuning for Language Models

Original Paper: https://arxiv.org/abs/2404.03592

By: Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, Christopher Potts

Abstract:

Large neural models are intended to be adjusted through updates to a limited number of weights using parameter-efficient finetuning (PEFT) techniques. Editing representations, however, may be a more effective option because a large body of previous interpretability research has demonstrated that representations encode significant semantic information.

In order to test this theory, we created a family of ReFT techniques. ReFT techniques learn task-specific interventions on hidden representations using a frozen base model.

We define Low-rank Linear Subspace ReFT (LoReFT), a strong instance of the ReFT family, and we find an ablation of this technique that gains efficiency at the expense of performance. Both learn interventions that are 15x–65x more parameter-efficient than LoRA and can be used as drop-in replacements for current PEFTs.

We demonstrate LoReFT on four arithmetic reasoning problems, instruction-tuning, GLUE, and eight commonsense reasoning tasks. Our ReFTs consistently surpass state-of-the-art PEFTs and offer the best efficiency/performance ratio in all of these studies.

A generic ReFT training library is made available to the public at [this https URL] (https://github.com/stanfordnlp/pyreft).

Summary Notes

Figure: Parameter count against performance for LoReFT and various PEFTs applied to LLaMA, Llama-2, Llama-3, and RoBERTa models over four benchmarks. On all tasks, LoReFT delivers comparable or even state-of-the-art performance, despite training much fewer parameters than existing PEFTs. In our evaluations, its worth is particularly noticeable for the biggest models.

Note that FT is not a PEFT or ReFT approach; rather, it is full-parameter finetuning.

Language models are becoming an essential tool for a wide range of applications in the rapidly changing field of artificial intelligence. Nonetheless, a crucial problem continues to be the difficulty of effectively tailoring these massive models to particular applications.

Using traditional methods requires complete finetuning, which takes a lot of time and resources. Presenting Representation Finetuning (ReFT), a novel method that focuses on the representations of the model instead of its weights, providing a more effective substitute.

We will explore the novel approaches and encouraging outcomes of ReFT in this blog post, especially its flagship application, Low-rank Linear Subspace ReFT (LoReFT).

Overview of ReFT

The ReFT method is based on the realization that language models hidden representations intrinsically encode rich semantic information.

This means that we can successfully steer model behaviors without changing model weights by interfering directly on these representations.

This idea differs from the conventional parameter-efficient finetuning (PEFT) techniques, which concentrate on changing a limited subset of the model weights.

LoReFT and other ReFT techniques work with a frozen basis model. They provide an effective substitute for current PEFTs by learning task-specific treatments on hidden representations.

The main idea is that more sophisticated and effective model adjustments are possible when representations are edited rather than weights.

Methodologies: The Operation of ReFT

Low-rank linear subspace interventions are used by ReFT. LoReFT specifically affects hidden representations in the linear subspace that a low-rank projection matrix defines.

The distributed alignment search (DAS) technique, which has been demonstrated to successfully locate concepts within model representations, provides the basis for this methodology.

The ReFT approach is broken down as follows:

• Low-Rank Projection: The main concept is to define a subspace inside the representation space of the model by means of a low-rank projection matrix. This subspace is then used for interventions.
• Task-Specific Interventions: ReFT can direct the model’s behavior during inference by figuring out which particular representations need to be changed in order to get desired results.
• Parameter Efficiency: LoReFT achieves state-of-the-art performance on many jobs with up to 65 times fewer parameters than classic PEFT methods like LoRA. This makes LoReFT incredibly efficient.

Key Findings and Results

ReFT, and especially LoReFT, have been shown to be effective on multiple benchmarks, indicating that they could be a strong substitute for current finetuning techniques. The highlights are as follows:

• Commonsense Reasoning: LoReFT outperformed all other techniques in accuracy and parameter efficiency, achieving state-of-the-art performance.
• Instruction-Following: LoReFT demonstrated its capacity to handle complicated, long-form text creation by outperforming both complete parameter finetuning and other PEFT algorithms in instruction tuning tests.
• Arithmetic Reasoning: Although LoReFT demonstrated competitive performance, it also made clear that more research is necessary to fully realize LoReFT’s potential in increasingly challenging reasoning tasks.
• Natural Language Understanding: LoReFT demonstrated adaptability to various model sizes and tasks, as seen by its performance on the GLUE benchmark, which was on par with the most effective PEFT approaches.

Applications and Implications

ReFT has significant ramifications and offers a fresh perspective on model finetuning:

• Resource Efficiency: ReFT minimizes the computational and memory cost related to fine-tuning big models by significantly lowering the number of parameters that require updating.
• Scalability: Techniques like ReFT that provide scalable finetuning solutions will become more and more crucial as models continue to expand in size.
• Versatility: ReFT is a desirable choice for deploying models in a variety of applications without requiring a significant amount of retraining due to its ability to perform effectively across a wide range of jobs.

Moreover, the ReFT architecture holds practical significance for the deployment of AI systems in situations with limited resources, like edge computing platforms or mobile devices. It is not merely an academic endeavor.

Final Thoughts

ReFT and its offshoot LoReFT is a noteworthy development in the domain of model finetuning. Through representation editing instead of weight adjustment, ReFT provides a more effective, scalable, and flexible method for model adaptation.

Innovations like ReFT will be essential to ensure that these systems can be deployed effectively and efficiently as AI continues to permeate every element of technology and society.

This is not where the journey ends. In order to achieve even better economies and capacities, future study will examine the application of ReFT to additional model families, probe further into the causal mechanisms underlying it, and improve the methodology.

ReFT is a powerful reminder that, occasionally, less really is more in the field of artificial intelligence (AI), where every parameter matters.

Feel free to check out more blogs, research paper summaries and resources on AI by visiting our website.