Benchmarks · Safety, security & robustness
CyberGym
also: Cyber Gym
Can an AI agent, given only a vulnerability's text description and the affected codebase, reproduce the flaw — generating a proof-of-concept input that actually triggers the bug — rather than merely describing or patching it?
UC Berkeley (Sunblaze Lab / Berkeley RDI)Released 3 June 2025Live
CyberGym, built by UC Berkeley’s Sunblaze Lab (part of Dawn Song’s Berkeley RDI group), measures whether an AI agent can reproduce a real software vulnerability rather than just discuss it. Given a vulnerable codebase and a text description of the flaw, the agent must generate a proof-of-concept input; a run counts only if that input triggers the bug on the pre-patch build and stops working once the official patch is applied. The benchmark draws its 1,507 instances across 188 open-source projects from Google’s OSS-Fuzz corpus, which makes the tasks concrete and their fixes verifiable.
The task is hard: at the paper’s June 2025 release the strongest model-and-scaffold combinations reproduced only around a fifth of the instances. The authors also reported a striking side effect — running frontier agents over the benchmark turned up 34 previously-unknown vulnerabilities and 18 cases where a historical patch had left the bug partly exploitable — evidence that the same capability being measured has direct offensive and defensive uses.
The benchmark returned to attention in August 2026, when Zhipu reported that GLM-5.3 scored 84.5% — a figure it called the best published CyberGym result, and one it said arrived without training specifically for the task. Those numbers, and the competitor figures Z.ai cited alongside them, are the vendor’s own comparison rather than independently verified leaderboard results, and the gap between reproduction (CyberGym) and full exploitation (measured by the separate ExploitBench and ExploitGym benchmarks) remains wide.
The set
1,507 real-world vulnerability instances across 188 open-source projects, built on top of Google's OSS-Fuzz corpus. Each task hands the agent a vulnerable repository and a description of the flaw; the agent must produce a proof-of-concept input, scored pass/fail by whether that input triggers the bug on the pre-patch build but not on the patched build. At the paper's publication the strongest model-and-scaffold combinations reproduced only about a fifth of the instances; the authors report that CyberGym runs also surfaced 34 previously-unknown (zero-day) vulnerabilities and 18 incomplete historical patches.
Example
A vulnerability instance drawn from the OSS-Fuzz corpus: the agent receives the pre-patch source tree and a text description of the flaw and must generate an input that crashes the vulnerable build while leaving the patched build unaffected — success is judged automatically by exit code, not by the agent's explanation.github.com
Where it stands
An emerging capability rather than a solved one. In the June 2025 paper the best systems reproduced only about 20% of the 1,507 instances. By August 2026 Z.ai reported GLM-5.3 at 84.5%, calling it the best published CyberGym result — but that figure, and the competitor numbers cited alongside it, come from the vendor's own launch comparison and have not been independently verified on CyberGym's public leaderboard. Read the recent frontier figures as company-reported.
How the top score changed hands
Best reported score at each point the lead changed hands. Tap a marker for the entry.
- June 2025Best system at publication~20% (reproduction rate)In the CyberGym paper's own evaluation, the strongest model-and-scaffold combinations reproduced only about a fifth of the 1,507 instances.
- June 2026GLM-5.277.2% (reproduction rate)The predecessor's figure, per Z.ai's GLM-5.3 launch comparison; company-reported.
- August 2026GLM-5.384.5% (reproduction rate)Z.ai-reported; the company's own comparison, ahead of the Mythos 5 and GPT-5.6 Sol figures it cited, but not independently verified.
Current best: GLM-5.3 — 84.5% (reproduction rate) Z.ai's reported figure at the GLM-5.3 launch, which the company described as the best published CyberGym result. Company-reported and not independently verified; Z.ai's same comparison listed Claude Mythos 5 at 83.8% and GPT-5.6 Sol at 83.6%.
In the timeline · 2 entries
Zhipu releases GLM-5.3 with an unplanned jump in cyber capability
Built on the same unretrained base model as GLM-5.2, it scored 84.5% on the CyberGym vulnerability-discovery benchmark and had its weights withheld for roughly two weeks.
Models & capabilities · Security & misuse
Anthropic previews Claude Mythos, withheld from public release over cyber-offense capability
Anthropic reported the model wrote a working Firefox exploit in 181 of several hundred attempts, versus two for its predecessor Opus 4.6, and found a 27-year-old OpenBSD bug.
Safety & alignment · Security & misuse · Models & capabilities