Timeline

MBZUAI releases K2-Horizon, six fully open models

Weights, code, training data and logs for models from 0.9B to 375B parameters under Apache 2.0; a November technical report says the 36B-A4B model nearly matches a 32B dense one.

  • Models & capabilities
  • Open weights & ecosystem
  • Minor

The Institute of Foundation Models (IFM) at the Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) in Abu Dhabi announced K2-Horizon on 3 September, a set of six language models ranging from a 0.9-billion-parameter dense model to a 375-billion-parameter mixture-of-experts model with 23 billion active parameters, released with weights, code, training data and methodology. Its Hugging Face repositories show creation dates of 1 September, and the 375B model’s card lists an Apache 2.0 licence. The technical report, credited to the “K2 Team”, followed on 16 November; the card had given its expected date as the end of September.

At launch, Eric Xing, MBZUAI’s president and the founder of IFM, said meaningful AI progress “depends on the ability to examine, build upon, and improve the technology, not simply access it through an API”. MBZUAI’s press release called K2-Horizon “the largest fully open model release in AI history”, a claim made by the university itself.

The paper’s abstract says the models “lead or are competitive with strong open models of their size on agentic and coding benchmarks”. It describes two architectural changes: Mixture-of-Value Attention, which applies mixture-of-experts routing to the attention value projections, and a softplus attention gate. It reports that the 36B-A4B model, which uses both, nearly matches a 32B dense model trained on the same data with one-seventh the active parameters. The authors also describe “Uno” adapters that speed up decoding by up to 2.2 times; the press release had given roughly three times for what it called a diffusion-distillation technique, which may be the same method.

The release follows the LLM360 definition of open source, publishing intermediate checkpoints from each training stage, pre-training data, training logs and code. The authors say about 10 trillion pre-training tokens were synthetic, and that four of the models share a 22-trillion-token corpus, which allows controlled comparison across scale and sparsity. This entry draws on the abstract, the model card and launch coverage; the benchmark results are the authors’ own.