Benchmarks · Coding & software engineering
SWE-sweep
Given a real repository with many bugs in it at once and no hint of what or where they are, how many can a coding agent find and fix on its own?
Meta FAIR, with Harvard, the University of Washington and StanfordReleased 1 October 2026Live
Benchmarks such as SWE-bench give an agent one bug at a time, with the issue report that describes it. SWE-sweep removes the report. Built by a team led from Meta FAIR, with authors from Harvard, the University of Washington and Stanford, it hands an agent a real repository containing dozens of known bugs at once and asks it to find and fix as many as it can. The repositories are rolled back to a point in their history where the largest number of later-fixed bugs coexisted, so every bug is real and has a test that proves the fix.
The scores are very low. When the paper was released on 1 October 2026, the best of seven models, GPT-5.6 Sol, fixed 4.7% of the 4,068 bugs. GPT-6 Astra, added later, reached 9.5%, at a cost of more than $3,000 for one run. The authors run every model through the same minimal agent, mini-SWE-agent; their paper reports that commercial agent tools such as Claude Code and Codex did not clearly do better.
Adoption is still early. As of late October the authors were the only ones publishing scores. Google’s Gemini 4 Carbon launch reported 9.6% from its own run, just above Astra, but described its metric as a subtask-weighted reward. That term does not appear in the benchmark’s own materials, so the two figures may not be directly comparable.
The set
100 open-source repositories in 22 programming languages, each checked out at a commit chosen so that as many known, later-fixed bugs as possible coexist: 4,068 reference bugs in all, from projects such as pandas, scikit-learn, NumPy, OpenCV and Lean 4. The agent gets no issue text, file names or bug types. A bug counts as fixed if its hidden tests pass and the original test suite still passes; a regression in the original suite scores the whole repository zero. The headline figure is the share of all 4,068 bugs fixed. Every board run uses the same minimal open-source agent, mini-SWE-agent.
Where it stands
New in October 2026 and far from saturated: the best model on the board fixed under a tenth of the bugs. So far only the authors run it — the board was finalised for an ICLR submission and public submissions were not yet open — and Google's Gemini 4 Carbon launch was the first lab table to cite it.
How the top score changed hands
Current best: GPT-6 Astra — 9.5% SWE-sweep board, read 29 October 2026 (mini-SWE-agent, $3,255 for the run). GPT-5.6 Sol (xhigh) 4.7%, Claude Opus 5.5 (xhigh) 3.6%, Claude Fable 5.1 2.9%. Google reported 9.6% for Gemini 4 Carbon from its own run, described as a 'subtask-weighted reward' — a term the benchmark's site and paper do not use — and Carbon is not on the board.