EleutherAI releases GPT-J-6B

EleutherAI released GPT-J-6B, a 6-billion-parameter language model trained on the Pile, the open text corpus the collective had assembled the previous year, and published the weights under an Apache 2.0 licence. It followed the group’s earlier GPT-Neo checkpoints and was, at release, the largest publicly downloadable GPT-3-style model.

The model was trained using Ben Wang’s Mesh Transformer JAX codebase on TPU infrastructure, and used rotary position embeddings rather than the learned positional embeddings of earlier GPT models. On the LAMBADA benchmark, a test of predicting a word that requires tracking context across a passage, EleutherAI reported GPT-J-6B scoring close to OpenAI’s similarly sized 6.7-billion-parameter GPT-3 model — evidence that a volunteer group without a commercial lab’s resources could match a comparably scaled proprietary model on at least one measure, even as GPT-3’s own largest, 175-billion-parameter version remained far out of reach.

Unlike GPT-3, which OpenAI made available only through a paid, waitlisted API, GPT-J-6B’s weights could be downloaded, inspected, fine-tuned and run by anyone with sufficient hardware. That openness cut both ways in reception: researchers and hobbyists used it as a base for fine-tuned chatbots, writing tools and research projects that would otherwise have depended on API access, but it was also adopted, without EleutherAI’s involvement, as the foundation for a bot fine-tuned on years of 4chan’s “Politically Incorrect” board posts and connected to the site the following year, generating large volumes of offensive content before it was taken down.

GPT-J-6B became one of the most widely used open-weight models of its period, cited as a baseline in subsequent research and folded into libraries such as Hugging Face’s Transformers, and it set up the pattern EleutherAI extended with GPT-NeoX and later Pythia: publishing the largest openly licensed models available at each point while commercial labs kept their frontier weights closed.