AI coding agents used to mass-check whether 2,200 ICML papers reproduce

Hugging Face reported results from a 19-day open reproducibility challenge in which 1,221 volunteers used AI coding agents — including Claude Code, Codex, Cursor, Pi and OpenResearch’s orx — to test claims made in papers accepted to ICML 2026. Participants examined 2,226 of the conference’s roughly 6,350 accepted papers, about a third of the total, and logged individual judgements on 35,908 specific claims between them.

The agents were tasked with reading a paper’s stated claims, writing code to reproduce the reported experiment, running it, and reporting whether the result held up. Across the papers checked, 51% had at least one claim independently verified through a real rerun of the experiment, while 23% had at least one claim the volunteers judged falsified or contested. Problems participants surfaced included mathematical proofs with undetected errors, theoretical claims with working counterexamples, mismatches between a paper’s stated method and its actual implementation, and evaluation metrics inflated through what the organisers described as padding.

Hugging Face framed the exercise as a demonstration of what large-scale, agent-assisted verification can add to peer review, but its write-up also emphasised the limits of automation: participants found that agents were reliable at executing experiments but that human oversight remained necessary to steer agents toward the right checks, question underlying assumptions and make judgement calls — such as evaluating whether a result matched a paper’s claim in spirit, not just in output — that current systems could not consistently make on their own. The challenge ran from 15 July to 2 August 2026, with results published on 13 August.