AI2 releases OLMoE mixture-of-experts model

The Allen Institute for AI released OLMoE, a sparse mixture-of-experts model with 1 billion active and 7 billion total parameters, trained on 5 trillion tokens jointly with Contextual AI. It extended AI2’s OLMo family — built around releasing every component of a model rather than weights alone — to the mixture-of-experts architecture other labs, including Mistral and DeepSeek, had already used to cut inference cost at a given quality level.

AI2 released not just the final weights but the pretraining data, code, evaluation scripts, training logs, and 244 intermediate checkpoints saved roughly every 5,000 training steps — the full record needed to reproduce or audit the training run, which almost no other lab of comparable output published alongside a release. The company reported the model trained roughly twice as fast as an equivalently capable dense model and placed it on what it called the Pareto frontier of performance against size, meaning no smaller open model it compared against matched its benchmark results. Post-training combined instruction data, mathematics and code with a preview of AI2’s Tulu 3 pipeline and direct preference optimisation.

OLMoE mattered less for its raw capability than for what it demonstrated: that the efficiency gains labs like Mistral and DeepSeek had shown from mixture-of-experts routing did not require withholding the training recipe, and that a fully open project could stay within a step or two of frontier efficiency techniques rather than trailing years behind them.