'Grokking' paper documents sudden generalisation long after overfitting

Researchers at OpenAI — Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra — published a paper describing “grokking”: small neural networks trained on simple algorithmic tasks (such as modular arithmetic) could sit at a memorised, overfit solution for a very long time and then, with no change in method, jump abruptly to near-perfect generalisation on held-out data.

The paper used deliberately toy datasets — binary operation tables such as division and permutation composition — precisely because their small, well-defined structure made it possible to watch training dynamics closely rather than infer them indirectly from a large language model’s behaviour. The authors reported that smaller datasets required substantially more optimisation steps before the generalisation jump occurred, and that the transition could arrive long after training loss had already bottomed out, i.e. long after the standard signal for “the network has finished learning.”

The finding cut against a common assumption: that a network which has memorised its training set is stuck at memorisation, and that overfitting and generalisation are separate regimes reached by separate networks. Grokking showed the same weights could pass through both, which reopened the question of what a network’s internal representation is actually doing during the long, apparently flat stretch of training that precedes the jump.

The paper’s immediate significance was modest — the tasks were toy, and the models involved were far smaller than contemporary language models. But it seeded a durable sub-literature in mechanistic interpretability, most notably Neel Nanda and colleagues’ 2023 paper “Progress measures for grokking via mechanistic interpretability,” which reverse-engineered what the grokked networks had actually learned (in that case, a Fourier-based algorithm for modular addition) and used the finding to argue that generalisation could be tracked by hidden metrics well before it showed up in test accuracy.