Anthropic publishes 'Toy Models of Superposition'

Nelson Elhage, Christopher Olah and fourteen co-authors at Anthropic published “Toy Models of Superposition,” a paper formalising why individual neurons in a neural network rarely correspond to single, human-interpretable concepts. Using small networks trained on synthetic data with a known, controllable set of underlying features, the authors showed that when a network has fewer neurons than there are features useful for its task, and those features are sparse — each rarely active at once — the network learns to represent more features than it has dimensions by packing them into overlapping, non-orthogonal directions. They called this compression “superposition.”

The paper worked through when superposition does and does not occur: it is favoured by sparsity in the underlying features and traded off against interference between them, and the authors showed the transition between a network representing few features cleanly and many features in superposition could itself be abrupt, following mathematical structure related to sphere-packing problems. Because a feature in superposition is not tied to one neuron but distributed across many, in combinations that also encode other unrelated features, standard interpretability techniques that inspect individual neurons cannot reliably say what the network is representing at any point.

The paper’s significance was diagnostic rather than a solution: it explained, with a controlled example rather than an unwieldy production model, why mechanistic interpretability work — including Anthropic’s earlier study of induction heads — kept running into representations that resisted clean decomposition. It framed the problem later interpretability techniques, including sparse autoencoders, were built to address: recovering something closer to a model’s true, disentangled features from its superposed internal activations.