Benchmarks · Safety, security & robustness
SEC-bench
also: SEC-Bench
Can an LLM agent handle a real software security task — reproducing a vulnerability and then patching it — in an authentic, containerised codebase?
University of Illinois Urbana-Champaign & PurdueReleased 13 June 2025Live
Most coding benchmarks ask a model to fix a bug once someone has already told it where to look. SEC-bench asks something closer to what a security engineer actually does: reproduce a real, disclosed vulnerability from scratch with a working proof-of-concept input, then write a patch that closes it without breaking the project. Researchers at the University of Illinois Urbana-Champaign and Purdue built the dataset with an automated pipeline of their own — a multi-agent system that pulls vulnerabilities from public disclosure databases, constructs a containerised, reproducible version of the affected codebase, and generates a reference patch, at a reported cost of under a dollar per instance.
Agents including SWE-agent, OpenHands and Aider were then set loose on two tasks scored against that harness: proving the vulnerability is real, and fixing it. The results were sobering relative to general software-engineering benchmarks of the same period — the best systems managed only 18% success generating a working proof-of-concept and 34% patching a known one, a reminder that security work demands a different kind of reasoning than resolving a labelled GitHub issue.
SEC-bench is maintained with a public leaderboard and a Hugging Face dataset, and its authors followed it in May 2026 with SEC-bench Pro, a harder successor built from real vulnerabilities in the V8 and SpiderMonkey JavaScript engines and the Linux kernel — evidence that, as with most benchmarks in this record, an early version measuring a genuine capability gap became the baseline for a tougher one once frontier models started closing it.
The set
An automated pipeline builds the dataset itself: a multi-agent system pulls real vulnerabilities from disclosure databases, constructs a reproducible containerised harness for each one, and generates a reference patch, at a reported cost of about $0.87 per instance. Agents such as SWE-agent, OpenHands and Aider are then scored on two tasks — proof-of-concept generation and vulnerability patching — against those harnesses.
Example
A real, disclosed vulnerability in an open-source project, handed to an agent inside a containerised copy of the codebase with the task of first reproducing the bug with a working proof-of-concept input, then producing a patch that fixes it without breaking the project's own tests.arxiv.org
Where it stands
At publication, the best agents reached only 18% success on proof-of-concept generation and 34% on patching, well below the saturation seen on general coding benchmarks — the harder SEC-bench Pro followed in May 2026.