Anthropic open-sources Bloom, an automated behavioural evaluation tool

Anthropic open-sourced Bloom, a tool that automatically generates diverse test scenarios to quantify how often and how severely a specified model behaviour occurs, released on GitHub under safety-research/bloom. The tool addresses a familiar problem in evaluation work: hand-built test sets are slow to construct and go stale as models change, while Bloom instead runs a repeatable four-stage pipeline — understanding a described behaviour, generating scenarios for it, rolling those scenarios out at scale with simulated users and tools, and judging the resulting transcripts with an AI judge and a synthesising meta-judge.

Anthropic benchmarked four behaviours — delusional sycophancy, instructed long-horizon sabotage, self-preservation and self-preferential bias — across 16 frontier models. It reported that automated scores for Claude Opus 4.1 correlated with human judgments at 0.86 on the Spearman measure, and noted that increased reasoning effort reduced self-preferential bias in Claude Sonnet 4, with the model more often recognising a conflict of interest rather than simply choosing itself less often.