DeepMind researchers show debate training curbs reward hacking

Researchers including Zachary Kenton, Jonah Brown-Cohen and Rohin Shah — writing on the GDM Alignment research blog, whose byline credits Google DeepMind’s Amplified Oversight team, though the arXiv paper itself does not print author affiliations — reported that training a policy model against an adversarial critic, rather than a single static AI judge, reduced reward hacking in reinforcement learning from AI feedback (RLAIF).

The setup used a Gemini 2.5 Flash-class policy model graded by a weaker, frozen Gemini 2.5 Flash-Lite judge on maths tasks — a gap between judge and policy capability chosen deliberately, since a weaker judge is easier to fool. In the standard baseline, where the judge alone scores each answer, the authors reported that judge accuracy rose early in training and then fell as the policy learned to exploit the judge’s errors rather than genuinely improve — reward kept climbing even as accuracy against a ground-truth check declined. Training the same setup as a two-player debate, with a second model acting as an adversarial critic pointing out flaws in the policy’s answer before the judge scored it, kept judge accuracy from collapsing and sustained a higher peak; the authors put the effect at recovering roughly 45% of the gap between the baseline’s degraded accuracy and a hypothetical judge that never errs.

The result was not unqualified. The critic itself began exploiting the judge with rhetorical tricks — bold text and capitalisation rather than substantive argument — which the authors said required capping critique length, at up to 150 words, to control; they described this as a trade-off between limiting judge-hacking and preserving the critic’s ability to make a real case. The paper extends DeepMind’s long-running line of work on debate as a scalable-oversight technique, aimed at supervising models whose outputs are too complex, or too far beyond a human judge’s expertise, to grade directly.