Idiap Research Institute Proposes HyperMixer: A Competitive MLP-based Green AI Alternative to Transformers
Transformer architectures have in recent years advanced the state-of-the-art across a wide range of natural language processing (NLP) tasks, and vision transformers (ViT) are now increasingly applied in the field of computer vision. But these successes come with a cost, as transformers’ quadratic complexity over input length results in very high energy consumption, limiting their research and development and industrial deployment.
In the new paper HyperMixer: An MLP-based Green AI Alternative to Transformers, a team from Switzerland’s Idiap Research Institute proposes a novel multi-layer perceptron (MLP) model, HyperMixer, as an energy-efficient alternative to transformers that retains similar inductive biases. The team shows that HyperMixer can achieve performance on par with transformers while substantially lowering costs in terms of processing time, training data and hyperparameter tuning.
The team summarizes their main contributions as:
- A novel all-MLP model, HyperMixer, with inductive biases inspired by transformers.
