The Emergence of Mixtral 8x7B: Redefining AI Ethics & Performance
By Lucy Hur
A few months ago, Open AI CEO Sam Altman said that “it’s totally hopeless for any startup to compete with Open AI”. However, in December 2023, French company Mistral dropped a torrent link to their brand new Apache 2.0 license model, announcing their new language model called Mixtral 8x7b.
Mistral, based in Paris, is a rising venture capital firm that develops fast, open source and secure language models. Founded in 2023, they’ve raised over $650 million in funding, are valued at $2 billion, and are less than a year old. Mixtral 8x7b is Mistral’s latest creation, positioned as a fast, open-source, and secure language model. While not yet reaching the performance level of GPT 4.0, it outperforms GPT 3.5 and LLaM 2 on most benchmarks (see figure 1.0 below). It has a true open-source license Apache 2.0, allowing developers to modify and profit from the model with minimal restrictions. The model runs locally with open weights. This allows developers to modify and profit from the model with limited restrictions.
The problem is that Mixtral, similar to Llama, is highly uncensored and “aligned out of the box” (read more on AI alignment here). Censored models usually have some sort of embedded alignment which stops the model from providing unethical information, such as teaching you how to cook meth or make bombs. They aim to stay politically ethical and filter information.
So why would anyone develop or use an uncensored model? Alignment may interfere with valid use cases. For instance, consider an analogy created by Eric Hartford, who writes on uncensored models. Eric writes, “Consider writing a novel. Some of the characters in the novel may be downright evil and do evil things, including rape, torture, and murder . . . many aligned models will refuse to help with writing such content . . . just wanting to know “how” to build a bomb, out of curiosity, is completely different from actually building and using one. Intellectual curiosity is not illegal, and the knowledge itself is not illegal.”
LexicaArt founder Sharif Shameem tweeted, “The Mixtral model genuinely feels like an inflection point — a true GPT-3.5 level model that can run at 30 tokens/sec on an M1. Imagine all the products now possible when interference is 100% free and your data stays on your device.” Mixtral 8x7B can process a 32K token context window (which is enough for 50 pages of text or a 2-hour-long conversation) and supports multiple languages including English, French, German, Spanish, and Italian. It’s now openly available to the public on Hugging Face and BitTorrent, and anyone can locally use it on an app called LM Studio.
The Mixtral model not only pushes the boundaries of technological capability but also prompts us to reconsider the relationship between alignment and censorship. As AI advances at a fast pace, finding a balance between alignment and performance will become increasingly important to set clear ethical standards for the industry.
