ExploitBench

Earlier security benchmarks tended to score exploitation as pass or fail — did the target crash, yes or no. ExploitBench, built by researchers at Carnegie Mellon University, argues that misses most of the story: a crash is only the first step in a real attack, and the harder, more consequential steps come after. It breaks the process into a 16-stage capability ladder running from reaching and triggering vulnerable code, through building sandbox-escape and memory-corruption primitives, to arbitrary read and write access and finally full arbitrary code execution — and scores an agent by how far up that ladder it climbs, not just whether it reaches the top.

The initial release runs the ladder against 41 real bugs in Google’s V8 JavaScript engine, comparing agents given only the target environment against agents given adaptive coaching feedback and a native command-line harness. The pattern that emerged was consistent: publicly available frontier models routinely got as far as triggering a crash but rarely converted that into working code execution, while one privately tested frontier model reached full code execution on roughly half the bugs — evidence, the authors argue, that hardened, real-world exploitation is only now becoming a capability frontier models are starting to cross, rather than one they have already mastered.

ExploitBench’s code, containers and model transcripts are published openly, with a leaderboard maintained at exploitbench.ai, positioning it as a running instrument for tracking exploitation capability rather than a one-off study — a distinction that matters given how directly the same capability bears on both defensive research and offensive risk.