Benchmarks · Safety, security & robustness
Cybench
Whether an AI agent can autonomously find and exploit real security vulnerabilities, using the same professional Capture the Flag (CTF) format security researchers train on.
Stanford UniversityReleased 15 August 2024Live
Capture the Flag competitions are how security researchers train and prove themselves: a target system hides a hidden “flag” string behind a real vulnerability, and solving the challenge means actually finding and exploiting it, not just describing the bug in the abstract. Cybench, built at Stanford, imports 40 professional-level CTF tasks from real competitions into an agent benchmark, so a model’s cyber-offence capability is measured the same way a human hacker’s is.
The tasks span cryptography, web exploitation, reverse engineering, forensics and binary exploitation, and each is calibrated against the human record: the easiest took a competition team 11 minutes to solve, the hardest nearly 24 hours. At launch, the best-performing model, Claude 3.5 Sonnet, solved only 7 of the 40 tasks unguided — a result the authors read as reassuring in the near term but a marker to watch, since the same benchmark could track how quickly that ceiling moved.
Cybench runs an ongoing leaderboard as newer models are evaluated, including results drawn from official system cards and the independently run Holistic Agent Leaderboard, making it one of the standard reference points labs and researchers cite when discussing how close frontier models are to autonomous cyber-offence capability — a question that, unlike raw coding ability, carries direct dual-use risk.
The set
40 professional-level CTF tasks drawn from four real competitions, spanning cryptography, web security, reverse engineering, forensics and exploitation, with 17 of the tasks broken into guided subtasks for gradated scoring. Difficulty is calibrated against how long the task took the human team that first solved it in competition, from 11 minutes up to nearly 24 hours. Agents are scored on unguided success, subtask-guided success, and the hardest task solved.
Example
A cryptography task supplies an encrypted artefact and flawed cryptographic code, asking the agent to identify the implementation weakness and recover the hidden key or plaintext; guided variants break this into subtask questions covering the same category as web, reverse-engineering, forensics and binary-exploitation tasks.
Where it stands
Maintains a running leaderboard as new frontier models are evaluated, alongside a separate track hosted on the Holistic Agent Leaderboard (HAL); this file reports only the launch paper's figures, which are now well behind current frontier performance.