OpenAI introduces SWE-bench Verified

OpenAI released SWE-bench Verified, a 500-task subset of the original SWE-bench coding benchmark, selected and validated by human software developers after OpenAI concluded the original benchmark was unreliable for evaluating coding agents.

The company ran 1,699 of SWE-bench’s test samples past 93 professional Python developers, who checked whether each problem’s description was clear and whether the automated tests used to grade a fix were correctly scoped to the issue. The screen found substantial quality problems: OpenAI reported that 38.3% of samples had an underspecified problem statement and 61.1% had unit tests it judged unfair — either too narrow to credit a valid alternative fix or unrelated to the stated issue — with 68.3% of samples excluded overall once environment setup problems and other issues were included. The surviving 500 formed the Verified set. On it, OpenAI reported GPT-4o resolving 33.2% of tasks, roughly double the 16% score models had been achieving on the unfiltered original set with the same scaffold.

The exercise was as much a comment on benchmark construction as on model capability: a widely used evaluation had been overstating both the difficulty of the underlying tasks and the unreliability of the scoring for two years before anyone systematically checked it. SWE-bench Verified became the standard reference for coding-agent evaluation through the following two years, cited in system cards from OpenAI, Anthropic and others, until OpenAI itself stopped using it in early 2026, after an internal audit found frontier models could reproduce many “gold patch” fixes from memory.