OpenAI stops evaluating models on SWE-bench Verified
An OpenAI audit found most frontier models, including its own, could reproduce gold-patch fixes from memory, and that a majority of remaining unsolved tasks were themselves flawed.
- Benchmarks & progress
- Minor
OpenAI said it would stop reporting model performance on SWE-bench Verified, one of the most widely cited coding benchmarks, arguing the test no longer measured frontier coding progress. Frontier models, including OpenAI’s own, were clustering around similar high scores, and the company said its own audit found the remaining headroom was largely an artefact of the test rather than a genuine capability gap.
Two problems drove the decision. Leading models — OpenAI’s GPT-5.2, Anthropic’s Claude Opus 4.5 and a preview Gemini model among them — could reproduce the benchmark’s original fix patches or problem statements from memory with minimal prompting, having encountered them during training on the public code repositories the benchmark itself draws from. Separately, engineers who reviewed a sample of the hardest remaining problems found a majority were flawed: tests demanding specific function names or features never mentioned in the problem description, so apparent model failures often reflected broken tests rather than real limitations.
OpenAI pointed to SWE-bench Pro, a harder benchmark built by Scale AI from more diverse, less publicly exposed codebases, as a partial replacement, while noting scores dropped sharply on it compared with the old benchmark. It also said it was building internally authored evaluations, graded by trained human reviewers, as a longer-term alternative to public benchmarks vulnerable to being memorised.
The move was a rare instance of a frontier lab publicly retiring a benchmark on which it had reported strong results, adding a concrete case to a broader pattern of coding and reasoning benchmarks losing usefulness once training data absorbed them.