OpenAI publishes 'Weak-to-Strong Generalization' superalignment paper

OpenAI’s Superalignment team, formed five months earlier to work on the technical problem of aligning models that might eventually exceed human capability, published its first major research output. The paper addressed a problem the team’s founding announcement had described only abstractly: if a future AI system were genuinely smarter than the humans training it, how could those humans supervise it, given that effective supervision usually requires the supervisor to be able to check the answer?

The researchers tested a proxy for that situation using models available today. They fine-tuned models from the GPT-4 family using labels generated by a much weaker supervisor — as weak as GPT-2 — instead of the higher-quality human labels normally used, then measured how much of the strong model’s own latent capability survived training on a weak teacher’s imperfect signal. Naive fine-tuning on GPT-2-level labels recovered a meaningful share of GPT-4’s performance on standard NLP benchmarks, and — critically — the resulting model consistently outperformed the weak supervisor it had learned from, evidence that some of the strong model’s own knowledge “leaked through” the weak signal rather than being capped by it. Adding an auxiliary confidence loss, designed to encourage the strong model to trust its own confident predictions even when they contradicted the weak label, pushed performance closer to what full fine-tuning on high-quality labels would have achieved, recovering close to GPT-3.5-level performance.

The method’s limits were reported alongside the successes: the same technique struggled to close much of the gap on harder tasks, including chess puzzle-solving and OpenAI’s internal reward-modelling dataset, suggesting that reinforcement learning from human feedback — the dominant alignment technique in production — might not straightforwardly scale to models whose outputs humans could no longer reliably judge. OpenAI framed the paper as a first, deliberately simple attempt at making the problem empirically tractable rather than purely philosophical, a research direction that outlasted the team itself: the Superalignment group dissolved five months later, in May 2024, after co-lead Jan Leike resigned citing insufficient compute and support.