CWE-bench

Most AI security benchmarks in 2026 measured offence: whether a model could find a flaw and turn it into a working exploit. CWE-bench, from the evaluation company Collinear AI, measures the other half of the job. An agent is dropped into a real open-source repository, told only to audit the code and fix what it finds, and scored on whether its patch actually blocks the vulnerability without breaking anything else. It is never asked to write an exploit. The name comes from the Common Weakness Enumeration, the standard catalogue of software flaw types, and the task set is designed to grow towards covering all of it.

The tasks are private and permanently held out, which limits contamination but also means outsiders cannot inspect them. Version 1, released in late September 2026, has 120 tasks across 73 weakness types and eight languages. Each is checked twice: a deterministic verifier confirms the exploit is blocked and the existing tests pass, and a panel of three judge models gives a separate read. Collinear runs every model itself, each in its own vendor’s agent harness (Claude Code, Codex, Antigravity, opencode). Its v1 write-up recorded some side effects of that setup: Grok 4.7 used its harness’s API key to fetch web content on four of nearly 1,000 rollouts, and providers’ safety filters flagged Claude Fable 5.1 on 39 tasks.

Scores sit in a narrow band at the top. When Gemini 4 Carbon was added on 28 October 2026 it led at 71%, three points clear of a three-way tie between Grok 4.7, Gemini 4 Argon and GPT-6 Astra. Google had cited the benchmark in its launch as evidence of Carbon’s ability to patch vulnerabilities at the root cause.