'Let's Verify Step by Step' introduces process supervision for reasoning

OpenAI researchers published “Let’s Verify Step by Step,” comparing two ways of training a reward model to check a language model’s mathematical reasoning: outcome supervision, which grades only the final answer, against process supervision, which grades each intermediate step of a solution. The paper reported that process supervision produced significantly more reliable reasoning, with a process-supervised model solving 78% of problems on a representative subset of the MATH benchmark.

To train the step-level reward model, the authors released PRM800K, a dataset of 800,000 human-generated labels judging the correctness of individual reasoning steps across tens of thousands of solutions — a considerably more expensive form of supervision to collect than a single right-or-wrong label per problem, since it required annotators to evaluate each step of often-long chains of reasoning. The paper also found that active learning, selectively sampling which solutions to have labelled, made this expensive process supervision more efficient to collect.

The core finding — that a model rewarded for the legitimacy of its reasoning path, not merely for arriving at a correct destination, made fewer errors and generalised its accuracy better — ran against the more common practice of training on final-answer correctness alone, and against the intuition that outcome-based reward should eventually be sufficient given enough data. It anticipated a shift the field made more fully the following year: training models with reinforcement learning against verifiable step-by-step reasoning became the technical basis for OpenAI’s o1 reasoning-model line and the wave of “thinking” models that followed it from other labs, which extended test-time compute specifically to let a model reason through more, and more carefully checked, intermediate steps before answering.