SEC-bench Pro

SEC-bench showed that agents struggled to patch known, disclosed vulnerabilities. SEC-bench Pro, released a year later by an overlapping team, raises the bar again: instead of a contained bug report, a model gets a real, large, actively maintained codebase — V8, SpiderMonkey or the Linux kernel — and has to work through its internals unassisted to reproduce a genuine vulnerability with a working exploit input. The 344 tasks span memory-safety bugs, sandbox escapes, JIT flaws, race conditions and kernel-subsystem issues, and each is validated against a real disclosure rather than synthesised.

The task is deliberately long-horizon: finding and proving a vulnerability in a codebase the size of a browser engine or an operating system kernel typically takes a human researcher days, and requires navigating build systems, debugging tools and unfamiliar subsystems rather than editing a single flagged function. Against that bar, the best system in the paper’s evaluation — OpenAI’s Codex running GPT-5.5 — reached 58% success, well ahead of open-weight alternatives, which lagged far behind; Anthropic’s Claude Code running Opus 4.6 solved most of the instances it completed but was frequently cut off by timeouts on the hardest cases.

The exercise had a real-world payoff: running the benchmark surfaced genuine, previously unknown vulnerabilities in V8 and SpiderMonkey, including a sandbox escape that the researchers reported through Google’s bug bounty programme and were paid $20,000 for. That outcome — a benchmark run turning up an actual, bounty-worthy security flaw — is one of the clearer pieces of evidence in this record that frontier models’ security capabilities have moved from academic exercise to a genuine, double-edged tool for offensive and defensive research alike.