Mistral releases Mixtral 8x7B

Mistral AI released Mixtral 8x7B, a sparse mixture-of-experts model, the same way it had released Mistral 7B two months earlier: with no announcement post, just a magnet link to the weights dropped on social media, followed by a licence — Apache 2.0, permitting unrestricted commercial use — and technical detail supplied afterward by the community and by Hugging Face’s own writeup.

The architecture replaced the feed-forward blocks in a standard transformer with eight expert feed-forward networks, routing each token through two of the eight at every layer. Despite the “8x7B” name, the model held roughly 45 billion total parameters rather than eight times seven billion, because only the feed-forward blocks were replicated per expert while other components, including attention, were shared. At inference the sparse routing meant it used far fewer active parameters per token than its total size would suggest, giving much of the speed of a smaller dense model.

On the benchmarks Hugging Face reported, Mixtral’s instruction-tuned version scored close to GPT-3.5-turbo on MT-Bench — 8.30 against 8.32 — and its base model outperformed Llama 2 70B on the standard open LLM leaderboard despite using a fraction of that model’s active compute per token. Hugging Face called it the first open model to reach GPT-3.5-level performance on that measure.

Mixtral’s release was read as evidence that mixture-of-experts architectures, until then mostly a closed-lab technique associated with unconfirmed reports about GPT-4’s own design, worked for openly released models too, and that a small, well-funded European lab could ship near-frontier capability months after its founding. It reinforced the argument, already building around Mistral 7B and Llama 2, that the gap between the best open-weight and closed-weight models could be closed faster than the labs restricting their own weights had suggested.