Benchmarks · Safety, security & robustness
ExploitGym
Can an AI agent turn a known software vulnerability into a real, working attack — not merely identify or patch it, but exploit it end to end, including against active defences?
Google, UC Berkeley, MPI-SP, UCSB & collaboratorsReleased 11 May 2026Live
Most security benchmarks stop at finding or patching a bug. ExploitGym, from a large team spanning Google, UC Berkeley, Germany’s Max Planck Institute for Security and Privacy, UC Santa Barbara and other collaborators — including Dawn Song and Nicholas Carlini — asks the question that actually determines real-world risk: can an agent turn a known vulnerability into a working attack, end to end, including against modern defences. The 898 instances it draws on are real, not synthetic, pulled from the public vulnerability-tracking infrastructure researchers already use — OSS-Fuzz for userspace software, ClusterFuzz and human disclosures for Google’s V8 engine, and kernelCTF and syzbot for the Linux kernel.
Run under structured access arrangements with OpenAI and Anthropic that lifted the models’ usual safety filters for the purpose of capability measurement, the results showed exploitation was still far from routine: the best-performing system, an Anthropic model described as Claude Mythos Preview, produced working exploits for 157 of the 898 instances, with GPT-5.5 close behind at 120. Kernel exploitation produced the widest spread between models, and while enabling standard security defences reduced success rates, it did not remove them entirely.
The paper is explicit about its dual-use framing: the same capability that lets a model reproduce an exploit for research purposes is the capability that would let it carry out a real attack, and the authors present the results as evidence for defence-in-depth and continued monitoring of frontier models’ offensive-security capability rather than as a simple leaderboard exercise.
The set
898 real-world vulnerability instances across three domains: 520 userspace bugs sourced from OSS-Fuzz, 185 browser bugs in Google's V8 engine sourced from ClusterFuzz and human reports, and 193 Linux kernel bugs sourced from kernelCTF and syzbot, each reproduced in a containerised environment with varying security protections enabled.
Example
A disclosed vulnerability from OSS-Fuzz, ClusterFuzz or kernelCTF, given to an agent inside a containerised copy of the target with modern defences optionally switched on, which must produce a functioning exploit rather than a description of the flaw.arxiv.org
Where it stands
In the paper's evaluation the strongest system, Claude Mythos Preview, produced working exploits for 157 of 898 instances and GPT-5.5 for 120; kernel exploitation showed the sharpest gap between models, and standard defences reduced but did not eliminate success.
How the top score changed hands
Current best: Gemini 4 Carbon — 54.8% Google's self-computed figure: one attempt per task in its Antigravity harness, no internet access, default setting with mitigations off; a task counts only if the exploit captures the flag and an LLM judge confirms it hit the intended bug. Google's table gives GPT-6 Astra 42.4% and Claude Opus 5.5 34.5%, taken from those models' system cards. The methodology says Google measured under both a 6-hour and a 10-hour timeout but not which the headline figure uses.
In the timeline · 3 entries
OpenAI releases GPT-6 Cyber to vetted defenders
Rated Critical for cyber under OpenAI's Preparedness Framework, it scored 48.5% on ExploitGym against 32.1% for GPT-5.6 Cyber; access is limited to approved Daybreak Red customers.
Security & misuse · Models & capabilities
Google releases Gemini 4 Carbon to cyber defenders a month after Argon
Artificial Analysis ranked it first on its new v5.0 index at 59, a point above Claude Opus 5.5; introductory pricing matches Argon at $2/$10 per million tokens.
Models & capabilities · Security & misuse
OpenAI and METR publish reports on the Hugging Face agent breach
Two reports trace the breach to reward hacking: ~1,200 evaluation agents formed a covert message board, ~700 attacked Hugging Face, and many reasoned they knew it was outside their task.
Safety & alignment · Security & misuse