LongBench

LongBench, built by researchers at Tsinghua University, was among the first benchmarks to move long-context evaluation beyond simple fact retrieval. The original 2023 release bundled 21 datasets across six task types — question answering over one or several documents, summarisation, few-shot learning, synthetic tasks and code completion — in both English and Chinese, testing whether a model’s understanding held up over documents running to several thousand words rather than just whether it could find one sentence.

As context windows grew and models improved, the original set stopped discriminating between them, and its authors built a harder successor. LongBench v2, released at the end of 2024, replaced the original tasks with 503 multiple-choice questions — written by nearly 100 highly educated professionals — spanning contexts from 8,000 words up to 2 million, and covering harder categories including long-dialogue understanding and whole-codebase comprehension. To calibrate difficulty, the authors had human experts attempt the same questions under a 15-minute time limit and scored 53.7%; standard models answering directly reached only 50.1%, below the human baseline.

The one model in the launch paper to exceed the human score was o1-preview, at 57.7%, which the authors attributed specifically to its extended reasoning process rather than a larger context window — evidence that on genuinely hard long-context reasoning, thinking longer about a shorter effective read mattered more than raw window size. LongBench v2’s construction — deliberately harder than its predecessor and calibrated against a measured human baseline rather than an arbitrary ceiling — set a pattern other long-context benchmarks followed as the original single-needle tests saturated.