Mixture-of-Depths, a Dazzling New AI Breakthrough

Conditional Computing is Finally Here

Few times I’ve been more excited about a research paper, and let me tell you I read many.

Presented by Google Deepmind and going by the name of Mixture-of-Depths (MoD), this research paper has everything it takes to become a seminal piece for future generations of state-of-the-art models.

Its principle? Not all thoughts are made equal.

In other words, MoD models can dynamically allocate compute to each prediction, just like a human would, tackling one of the greatest issues our current frontier models have.

This probably won’t say much to you right now, but I guarantee you are going to be very excited by the end, as MoDs not only drastically reduce computing requirements to run models, but they present the opportunity to create smarter and more powerful ones.

And the best thing? This can be applied to every single LLM in the world.

Not all Thoughts are Created Equal

Humans, when faced with a task, can decide how much thought, or effort, they will dedicate to the problem.