DROP

Most reading-comprehension benchmarks of the late 2010s could be beaten by finding the right sentence and copying a span out of it. DROP was built specifically to close that shortcut: its 96,000 crowdsourced questions, drawn from Wikipedia passages, typically require pulling numbers or facts from several different places in a paragraph and then doing something with them — adding, counting, sorting, or comparing dates — rather than lifting an answer directly from the text.

The design worked. At launch in 2019, general-purpose reading models scored only 32.7 F1, and even a system built specifically to combine reading comprehension with numerical operations reached just 47.0, against a 96.0 F1 human ceiling. That gap closed unevenly as large language models arrived: OpenAI’s GPT-4 technical report cited 80.9 F1 for GPT-4 against 64.1 for GPT-3.5, a substantial jump, though still short of the 88.4 F1 the report attributed to systems trained specifically on the benchmark rather than evaluated zero- or few-shot.

DROP settled into the same role as several of its era-mates — HellaSwag, WinoGrande, ARC — a benchmark still run as one line in a broader evaluation table (AI2’s Olmo 3, for instance, cited DROP scores alongside SQuAD and HumanEval in 2025) but no longer a headline figure in a frontier release. Its multi-step arithmetic-over-text format was largely absorbed into broader reasoning and math suites once those became the more informative test of the same underlying skill.