Cerebras releases seven open Cerebras-GPT models

Cerebras released Cerebras-GPT, a family of seven GPT-style language models ranging from 111 million to 13 billion parameters, with weights, training data recipe and a paper describing the method all made public under the Apache 2.0 licence. The models were trained on the Andromeda supercomputer, a cluster of 16 of the company’s CS-2 wafer-scale systems, using a simple data-parallel approach the company said avoided the complex model-partitioning schemes large clusters usually require.

The release’s stated purpose was less the models themselves than the scaling-law study behind them. Every model in the family was trained following the Chinchilla recipe of roughly 20 training tokens per parameter, on the openly available Pile dataset, and Cerebras argued this made theirs the first published scaling-law analysis built entirely on public data and public checkpoints — earlier scaling studies from Google DeepMind and OpenAI had relied on proprietary training runs that outside researchers could not reproduce.

Cerebras framed the release as evidence for its wafer-scale hardware, released alongside the arXiv paper describing training efficiency and per-token cost. The models themselves were not intended to compete on capability with contemporaries such as LLaMA; at 111M-13B parameters and trained compute-optimally rather than beyond it, they sat below the frontier. Their contribution was procedural rather than a capability jump: a reproducible, fully open scaling-law dataset that other researchers could build on without access to a major lab’s internal training infrastructure.