MMLU

For most of the early 2020s, if a new language model was described in a single number, that number was its MMLU score. Dan Hendrycks and colleagues at UC Berkeley published the benchmark in 2020 as a deliberately broad test of knowledge: 15,908 multiple-choice questions across 57 subjects, from elementary arithmetic to professional law and clinical medicine, chosen so that doing well required genuine world knowledge rather than a single linguistic trick.

Its breadth was exactly what made it useful. A model’s average across all 57 subjects compressed a great deal of competence into one comparable figure, and MMLU appeared in essentially every frontier release from GPT-4 onward. That same breadth is what let it saturate: as leading models climbed into the high 80s and low 90s, they approached the accuracy ceiling set by errors in the questions themselves, and the gap between “state of the art” and “as good as the test can measure” closed.

The response was a familiar one in this record — build a harder version. The same group released MMLU-Pro in 2024, with more answer options and more reasoning-heavy questions, and the field’s attention moved on. MMLU’s arc — rapid adoption as the universal yardstick, then rapid saturation — became the template that later benchmarks, from GPQA to Humanity’s Last Exam, were explicitly designed to outrun.