EleutherAI announces GPT-NeoX-20B

EleutherAI, the volunteer research collective that had spent the previous year building open alternatives to GPT-3, announced GPT-NeoX-20B: a 20-billion-parameter autoregressive language model, trained in collaboration with CoreWeave, which it described as the largest dense model with publicly available weights at the time.

Checkpoints became downloadable from The Eye a week later, released under the permissive Apache 2.0 licence — no request forms, no gated access, no commercial restrictions. A hosted inference option through the GooseAI service was also announced for developers who did not want to run the weights themselves. EleutherAI’s own framing was cautious: it called the model “a research artifact” and advised against production deployment without further evaluation, noting that the training data — the Pile, EleutherAI’s own web-scale corpus — had received less filtering than commercial models typically applied.

The release mattered less for outright capability — GPT-NeoX-20B underperformed contemporary closed models on most benchmarks — than for what it made available. At the time, models above 10 billion parameters existed almost exclusively behind API paywalls at OpenAI, Google and a handful of others. A dense 20B model that anyone could download, fine-tune and inspect gave outside researchers a much larger open testbed than had previously existed, and it became a frequently cited baseline in fine-tuning and evaluation papers over the following two years. A companion technical paper detailing the full architecture and training methodology followed in April 2022.