OpenAI publishes 'Deliberative alignment' research

OpenAI described “deliberative alignment,” a training method used on its o-series reasoning models that has a model explicitly recall and reason over the text of a written safety policy before producing an answer, rather than relying only on reinforcement learning from human feedback to instil safe behaviour implicitly.

The method trains in two stages: supervised fine-tuning on chain-of-thought reasoning generated against the safety specification using synthetic data, followed by reinforcement learning that rewards adherence to the policy. OpenAI reported this needed no human-written example completions, reducing the labelling effort safety training usually requires. On the company’s StrongREJECT jailbreak benchmark, o1 scored 0.88 against 0.37 for GPT-4o, and it reached 93% accuracy on XSTest, a test of not refusing benign prompts that resemble unsafe ones. OpenAI also reported the approach generalised to attack types not seen in training, including prompts in other languages or encoded to evade filters.

The publication came on the final day of OpenAI’s “12 Days of OpenAI” event, alongside the announcement of o3, and set out the safety technique behind both models. It offered an answer, at least on OpenAI’s own benchmarks, to a criticism that had followed reasoning models since o1’s release: that a chain of thought hidden from the user was hard to audit for whether the model’s stated reasoning matched what actually drove its answer. OpenAI’s figures were self-reported and not independently reproduced.