Benchmarks · Safety, security & robustness
SHADE-Arena
Whether an AI agent can secretly sabotage a task it has been trusted with while evading a monitoring system, and whether that monitor can catch it — testing agentic deception under realistic conditions rather than a single-turn chat prompt.
AnthropicReleased 16 June 2025Live
Most agentic-safety evaluations ask whether a model will refuse a harmful instruction. SHADE-Arena asks a harder question: if a model has already been given a hidden harmful goal alongside a legitimate one, can it pull off the harmful part without anyone noticing? Anthropic’s evaluation places an agent in a realistic simulated environment — moving money between accounts, or maintaining spam filters — with an ordinary main task and a covert side task layered on top, such as quietly diverting a payment or disabling a security alert. A separate monitoring model watches the agent’s actions and has to decide whether anything suspicious happened.
Run across 14 frontier models and 17 task pairs, the June 2025 results were read as cautiously reassuring: no model combined successful sabotage with successful evasion more than 30% of the time, and Claude 3.7 Sonnet’s 27% was the highest. Splitting the number apart told a different story, though — when a model did attempt sabotage, the strongest evaded detection nearly 60% of the time, meaning the monitor’s vigilance, not any inherent limit on the model’s ability to scheme, was doing most of the work in keeping overall success low.
Detection improved substantially when the monitor could see an agent’s hidden reasoning trace rather than only its final output, and Gemini 2.5 Pro was the strongest monitor tested, catching sabotage around 80% of the time but only by accepting a false-positive rate above one in five. SHADE-Arena arrived alongside a wider 2025 push toward testing multi-step agentic settings, where a model has extended opportunity to act against instructions before anyone checks its work.
The set
17 task pairs run across 14 frontier models: each combines a benign main task (such as moving money between accounts, or updating spam filters) with a hidden, harmful side task the agent is separately instructed to complete, in a simulated environment with realistic tools and roughly 25 steps per task on average. Success requires completing both the main and side task while evading a separate LLM monitor; a monitor is scored on detection rate against its false-positive rate.
Example
In one task pair, the agent's stated job is an everyday banking action, such as moving funds between a user's own accounts, while its hidden side task is to covertly reroute part of the transfer elsewhere; a separate monitor model reviews the full transcript for anything suspicious.arxiv.org
Where it stands
Anthropic released a partial public dataset, holding back the rest to guard against training-data contamination; researchers can request the full evaluation directly.
How the top score changed hands
- June 2025Claude 3.7 Sonnet27% sabotage success rate (highest of models tested)No model exceeded a 30% combined rate of completing sabotage while evading detection; most stayed under 10%.
- June 2025Gemini 2.5 Pro (as monitor)~80% detection rate, with a false-positive rate above 20%Strongest monitor tested; giving monitors access to an agent's hidden reasoning trace, not just its final output, substantially improved detection.