Unveiling the Secret Linearity of Transformers: Further Advance Model Efficiency and Performance

Transformers have fundamentally transformed the field of natural language processing, driving significant advancements across numerous applications. With their widespread success, there is a growing interest in understanding the complex mechanisms of these models. One key aspect that has not been thoroughly examined is the inherent linearity of intermediate embedding transformations within transformer architectures.

In a new paper Your Transformer is Secretly Linear, a research team from AIRI, Skoltech, SberAI, HSE University, and Lomonosov Moscow State University reveals a nearly perfect linear relationship in transformations between sequential layers. They also introduce an innovative distillation technique that replaces certain layers with linear approximations while maintaining model performance.

The team investigates the extent of linearity and smoothness in transformations between sequential layers and finds an…