SWE-bench paper published

Researchers led by Carlos E. Jiménez and John Yang, working across Princeton and the University of Chicago, published SWE-bench, a benchmark built from 2,294 real software engineering problems drawn from GitHub issues and their corresponding merged pull requests across 12 popular Python repositories. Rather than isolated coding puzzles, each task required a model to read an actual bug report or feature request, locate the relevant code across a real, multi-file codebase, and produce a patch that a human reviewer had judged sufficient to resolve the issue and that passed the project’s own test suite.

Performance at publication was low. The best-performing model tested, Claude 2, resolved just 1.96% of the issues; a version of the open-weight Llama 2 fine-tuned specifically for the task, called SWE-Llama, did little better. The gap between this and models’ strong scores on narrower coding benchmarks such as HumanEval illustrated a distinction the paper’s authors argued mattered: writing a short, self-contained function from a specification is a different skill from finding and fixing a bug inside an existing, unfamiliar, real-world codebase, which requires navigation, context-gathering and multi-file reasoning that earlier benchmarks did not test.

SWE-bench’s low starting scores turned out to make it more useful, not less, as later coding agents were built specifically to close the gap: scores rose sharply over the following two years as labs and start-ups optimised for the benchmark directly, agent scaffolding matured, and models improved at long-horizon tool use. It became the dominant reference point for coding-agent capability through 2024 and 2025, cited in the release materials of most major coding-focused models, and prompted derivative and harder benchmarks once the original was judged close to saturated — including Scale AI’s SWE-bench Pro in 2025, built explicitly to restore headroom the original no longer had.