MosaicML releases MPT-7B
Four variants shipped together, including a version extrapolating to roughly 84,000 tokens of context — far beyond the 2,000-4,000 tokens typical of open models at the time.
- Open weights & ecosystem
- Models & capabilities
- Minor
MosaicML released MPT-7B, a 6.7-billion-parameter language model trained on 1 trillion tokens of text and code, licensed under Apache 2.0 for unrestricted commercial use. The company said the base model was trained in 9.5 days on 440 Nvidia A100 GPUs on the MosaicML platform, at a cost it put at roughly $200,000, and reported that it matched the quality of Meta’s LLaMA-7B — each model led on roughly half of twelve benchmark tasks the two were compared against — while being free of LLaMA’s non-commercial research licence.
MosaicML released four variants at once: the base model, an instruction-following version under a share-alike licence, a chat-tuned version restricted to non-commercial use, and MPT-7B-StoryWriter-65k+, built for long-form text with a context window extended to 65,000 tokens during fine-tuning and reported to extrapolate to roughly 84,000 tokens at inference — well beyond the 2,000-4,000-token windows typical of open models at the time.
The release landed the same week as Hugging Face’s StarCoder, part of a cluster of open-model releases through April and May 2023 aimed at giving developers commercially usable alternatives to closed APIs. MosaicML was acquired by Databricks the following month, which continued MPT development as part of its own open-model efforts.