A new benchmark targets flaws in SWE-bench, the standard coding-agent test

A group of researchers released SWE-Bench ProMax, a benchmark for testing AI coding agents on large-scale code refactoring across seven programming languages, built in direct response to reported quality problems in SWE-bench Verified, the benchmark most widely used to measure coding-agent capability. The paper cites an unspecified prior audit finding that nearly 60% of instances SWE-bench Verified marks as unsolved actually contain flawed tests — either overly strict tests that reject genuinely correct solutions, or overly permissive ones that pass on undocumented requirements the model was never told to meet. If that figure holds, a meaningful share of the scores labs have reported and competed over reflect test defects rather than model capability.

ProMax comprises 170 refactoring tasks drawn from real repository commits across Python, Java, TypeScript, Go, C, C++ and Rust, averaging more than 11 modified files and over 260 lines of changed code per task — deliberately larger and more multilingual than SWE-bench’s original Python bug-fix format. The authors rewrote issue descriptions for clarity and manually reviewed test suites to remove the kind of flawed tests they say undermine the older benchmark. On their own evaluation, frontier models resolved only about 41% of ProMax tasks, indicating the benchmark is not yet saturated.

SWE-bench Verified has functioned as a de facto leaderboard for coding agents since its release, cited by nearly every major lab in model announcements; a credible challenge to its test quality bears on how much weight those comparisons deserve, and on whether newer, harder benchmarks like ProMax will be adopted widely enough to replace it as the reference standard.