EleutherAI releases The Pile

Six months after forming to replicate GPT-3 in the open, EleutherAI released The Pile, an 825GiB English-language text corpus assembled by Leo Gao, Stella Biderman, Sid Black and collaborators for training large language models. The paper describing it appeared on arXiv on the last day of 2020.

The corpus was not another scrape of the open web. It combined 22 separate subsets — some newly constructed, others existing datasets such as PubMed, Books3, arXiv and GitHub — weighted toward sources with denser, more specialised knowledge than a general web crawl provides. The authors reported that models trained on the mixture generalised better across domains than models trained on raw Common Crawl text, with the largest gains on academic and technical material where GPT-2 and GPT-3 had previously performed weakly. They published the construction code alongside the dataset and documented, rather than hid, the composition’s known problems: it contained copyrighted books, pirated text sources, and content that made no attempt at deduplication or provenance filtering.

At the time, no comparably sized, openly documented training corpus existed outside a handful of frontier labs — GPT-3’s training data was described in general terms but never released. The Pile filled that gap and became one of the most widely reused datasets in open-source language modelling over the following several years, training runs that included EleutherAI’s own GPT-Neo and GPT-J as well as models built by other groups entirely. Its inclusion of Books3 and other copyrighted text later made it a focal point in the copyright lawsuits filed against AI developers, once plaintiffs’ lawyers established which training sets had drawn on it — turning a dataset built to democratise access into evidence in the argument over what that access had cost.