Gwern publishes 'The Scaling Hypothesis'

The pseudonymous writer Gwern Branwen published “The Scaling Hypothesis,” an essay written around GPT-3’s release that argued the model’s new abilities came not from any architectural innovation but simply from training a familiar Transformer design at far larger scale. GPT-3 had shown meta-learning behaviour — adapting to new tasks from a handful of examples in its prompt — that its similarly-designed but smaller predecessor, GPT-2, had not. Gwern’s argument was that this was not a coincidence: capabilities were emerging as a predictable, if not fully explained, consequence of scale on compute, data and parameters, following power-law curves that showed no sign of running out.

The essay framed this as a contrarian claim within machine learning, where clever, hand-designed architectures were traditionally valued over brute-force compute. It drew explicitly on Richard Sutton’s “bitter lesson,” the observation that methods which exploit computation had repeatedly beaten hand-engineered approaches across the field’s history, and coined its own term, “the blessings of scale,” for the pattern it described.

The blessings of scale is the observation that for deep learning, hard problems are easier to solve than easy problems — everything gets better as it gets larger (in contrast to the usual outcome in research, where small things are hard and large things impossible).

— Gwern Branwen, “The Scaling Hypothesis”

Gwern’s essay was not a research paper and made no new empirical claim of its own; it was a synthesis of evidence already visible in OpenAI’s and others’ published scaling-law work, argued with unusual force at a moment when many researchers still treated GPT-3’s results as a curiosity rather than a predictable trend. The essay was revised repeatedly through early 2022, before DeepMind’s Chinchilla paper complicated the compute-optimal picture it described. It became one of the most widely cited informal statements of the position that came to be called the scaling hypothesis, read inside and outside frontier labs as an argument for why scaling up existing architectures, rather than searching for new ones, was the more promising path to more capable systems.