BIG-bench paper released

More than 450 authors from 132 institutions published the paper describing BIG-bench (the Beyond the Imitation Game Benchmark), a collaboratively assembled collection of 204 tasks designed to probe capabilities existing language-model benchmarks did not cover — including tasks touching linguistics, mathematics, common-sense reasoning, biology, physics, social bias and software development. Contributors could propose their own tasks for inclusion, which is how the benchmark reached its scale and breadth: no single lab wrote the whole thing.

The paper’s central finding was that model performance and calibration both improved as models scaled up, but remained poor in absolute terms and well behind human rater performance on the same tasks, even for the largest models tested (up to hundreds of billions of parameters). Most tasks improved gradually and predictably with scale, in ways consistent with memorisation getting better; a smaller number showed “breakthrough” behaviour — near-random performance until a threshold, then a sharp jump — which the authors linked to tasks requiring several reasoning steps or components to work together. The paper also found social bias in model outputs tended to increase with scale in ambiguous contexts, though targeted prompting could reduce it.

BIG-bench mattered as an infrastructure project as much as a research result: by the time GPT-4 and its contemporaries arrived, standard benchmarks such as MMLU were saturating, and BIG-bench’s harder, more varied tasks — along with the harder BIG-bench Hard subset later extracted from it — became reference points in later arguments about whether headline benchmark scores still meant much. A companion entry in this record covers the benchmark’s public data-collection effort as a distinct milestone.