Paper finds flaws in about 30% of Terminal-Bench 2.0 tasks

E. Kelly Buchanan, Christopher Ré and colleagues, writing from Stanford and Cornell, posted a paper on arXiv that documents how the maintainers audited Terminal-Bench 2.0, the command-line coding-agent benchmark launched in 2025. The audit found issues in roughly 30% of the benchmark. The repaired set, Terminal-Bench 2.1, was not new on 5 November: the benchmark’s own blog index lists its release on 6 May 2026 as a revision that fixes 28 tasks, so the paper is the formal account of an existing version.

The authors argue that benchmarks accumulate “maintenance debt”: as models improve they find valid solutions that old verifiers reject, and changes to infrastructure break assumptions built into tasks. Their remedy is a continuous-validation workflow that combines community reports, task artifacts, agent traces, automated checks and human-approved repairs. They re-ran 19 agents on both versions. Accuracy rose by over 10 percentage points in the abstract’s summary, and leaderboard positions moved by up to three ranks, although the overall ranking stayed highly correlated (Spearman 0.958). The body notes the effect was uneven, and that the two weakest agents lost accuracy on 2.1, because some repairs made tasks stricter. Against the final 2.1 release, the audit workflow reached 57.1% precision and 71.4% recall; the authors caution that this measures agreement with a release the workflow helped build, not independent defect discovery.

A second finding concerns resources. Extending the time limit closed a 20-point gap between two harnesses running GPT 5.4, which the authors take to mean that time budgets are part of the measurement and not a neutral setting.

A separate audit appeared two days earlier. Prime Intellect reported that it rewrote request schemas and loosened some graders in Zapier’s AutomationBench, after which Claude Opus 5.5 scored 92.15% (70.83% on the strict measure). It said the fixes apply only to the public set, which scores higher than the held-out set, and that the results are not comparable with Zapier’s own reported mode. The company also described Databricks’ OfficeQA Pro v2 as brittle in its regex-based grader, and said that after its changes the benchmark is near saturation, with Opus 5.5 at 85.6%. Neither audit has been independently replicated, as far as the sources show.

The paper observes that leaderboards “collapse benchmark version, harness, and resource budget into one number”. Its argument is that a reported score is only comparable with another when all three match, which is the point on which Prime Intellect’s rescoring, with its warning about comparability, also lands.