Hendrycks et al. publish the MMLU benchmark

Dan Hendrycks and coauthors, including Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt, released the Massive Multitask Language Understanding benchmark, a set of 15,908 multiple-choice questions spanning 57 subjects — from elementary mathematics and US history to professional law and clinical medicine — designed to jointly test a model’s world knowledge and problem-solving ability rather than a narrow linguistic skill.

At the time of publication the paper tested existing models, including GPT-3, rather than the model MMLU would later become famous for grading. The largest GPT-3 variant improved over random guessing by nearly 20 percentage points on average across the 57 subjects but remained well short of expert-level accuracy, and the paper noted particular weakness on ethically and socially sensitive subjects such as morality and law — model performance was uneven across the suite rather than uniformly weak.

MMLU went on to become the field’s default general-knowledge benchmark, cited in essentially every frontier model release from GPT-3’s successors onward, because it was broad enough to summarise a model’s competence in a single reported number. That same breadth made it saturate: by the mid-2020s leading models were scoring in the high 80s and low 90s, close to the ceiling set by label noise in the original question set, prompting the same authors to publish the harder MMLU-Pro in 2024. MMLU’s trajectory — rapid adoption, then rapid saturation — became a template repeated by several benchmarks that followed it.