Anthropic publishes Constitutional AI

Anthropic published Constitutional AI, a paper led by Yuntao Bai proposing a way to train a “harmless” assistant without relying on humans to label individual harmful outputs. In a supervised phase, a model was prompted to critique its own responses against a written list of principles — the “constitution” — and revise them accordingly, then fine-tuned on its own revisions. In a reinforcement-learning phase, the model itself judged which of two candidate outputs better followed the constitution, and those AI-generated preference judgements trained a reward model used to fine-tune the policy — a process Anthropic termed reinforcement learning from AI feedback, or RLAIF, by analogy with the human-feedback method (RLHF) that had trained OpenAI’s InstructGPT the previous January.

The stated advantage was scalability: human oversight was reduced to writing and revising the list of principles, rather than reviewing large volumes of individual model outputs for harm, which Anthropic argued would not keep pace with model capability as systems grew larger and were asked to handle more varied situations. The method also made the standard the model was held to explicit and inspectable — a written document — rather than implicit in scattered human judgements.

Constitutional AI became one of the more consequential methodological choices in Anthropic’s approach to training its Claude models, and the constitution itself was later published and periodically revised as a public document. The RLAIF technique it introduced also anticipated a broader industry shift toward using models to supervise and evaluate other models — a scaling strategy for alignment work later formalised, for Anthropic, in its Responsible Scaling Policy.