METR: many SWE-bench-passing pull requests would not actually be merged

METR published a note reporting that a large share of pull requests marked “passing” on SWE-bench Verified — a widely used benchmark for AI coding agents, scored by an automated test suite — would not actually have been accepted by the human maintainers of the underlying open-source projects.

Four active maintainers from three repositories in the benchmark, scikit-learn, Sphinx and pytest, reviewed 296 AI-generated pull requests that had passed SWE-bench Verified’s automated grading. METR reported that roughly half would be rejected, most often because the change failed to actually fix the underlying issue, broke other functionality, or fell short of the project’s code-quality standards, despite satisfying the benchmark’s tests. To control for reviewers simply being harsh critics, METR also had them assess a “golden baseline” of human-written PRs from the same repositories under blinded conditions, of which a majority were judged mergeable — establishing a roughly 24-percentage-point gap between the benchmark’s automated pass rate and what maintainers would actually accept.

METR framed the result not as evidence of a fundamental limitation in coding agents but as a warning about reading benchmark scores literally: a “naive interpretation” of SWE-bench Verified results risks overstating how useful an agent’s output would be without the iteration and human feedback that real development affords, and that the benchmark’s automated grader was never designed to substitute for maintainer judgement. The finding added to a broader run of 2026 work questioning whether headline scores on saturating coding and agentic benchmarks reflect real-world reliability.