Hugging Face releases FineWeb dataset

Hugging Face released FineWeb, a roughly 15-trillion-token English web dataset built by cleaning and deduplicating text from 96 Common Crawl snapshots spanning 2013 to April 2024. The dataset was published under the ODC-BY licence and accompanied by full documentation of the filtering pipeline, in contrast to the largely undisclosed training corpora used by closed frontier labs.

Alongside the base dataset, Hugging Face released FineWeb-Edu, a smaller subset filtered for educational value using a classifier trained on scores that the model Llama-3-70B-Instruct assigned to half a million sample documents. The filtered subset was intended to improve performance on knowledge benchmarks such as MMLU without requiring the full, much larger corpus.

FineWeb was pitched as a fully open, fully reproducible alternative to earlier open web corpora such as C4 and RefinedWeb, both narrower and, Hugging Face argued, lower quality at comparable scale. For open-weight model developers without access to a proprietary web-scraping pipeline, it removed one of the largest remaining gaps between what closed labs could build training sets from and what was publicly available, and it became a widely used default for pretraining runs outside the largest labs.