Anthropic publishes 'Red Teaming Language Models to Reduce Harms'

Anthropic published a paper describing its early efforts to systematically red-team its own language models — having people deliberately try to provoke offensive, unethical or otherwise harmful output — and to measure whether that process could be used to make models safer rather than merely to catalogue their failures.

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs.

— Anthropic, “Red Teaming Language Models to Reduce Harms”

The team compared four kinds of model — a plain language model, one prompted to be helpful and harmless, one using rejection sampling, and one fine-tuned with reinforcement learning from human feedback (RLHF) — at three sizes ranging from 2.7 billion to 52 billion parameters. The central finding was that RLHF-trained models became harder to successfully red-team as they scaled up, while the other three approaches showed no such improvement with size — a flat trend regardless of how large the model got. That gave an early, quantitative argument that RLHF specifically, rather than scale alone, was doing the safety work.

Anthropic released the full set of 38,961 red-team attack transcripts collected during the study, along with detailed documentation of its instructions, statistical methods and uncertainties, framing the release as a step toward transparency in a research area where labs had mostly kept their internal red-teaming private. The dataset became a widely used reference in later safety and jailbreak research, and the paper’s comparison of training methods fed directly into the case for RLHF as a standard step in frontier model development.