Announcing ARC Prize 2024

François Chollet, the Keras creator who introduced the Abstraction and Reasoning Corpus (ARC-AGI) in 2019 as a benchmark that resisted memorisation, and Zapier co-founder Mike Knoop launched ARC Prize 2024, a public competition offering a prize pool of more than $1 million to any team that could substantially advance the state of the art on it. Each ARC-AGI task presents a small number of input-output grid examples and asks a solver to infer the underlying rule and apply it to a new grid — designed so that humans find the tasks straightforward while systems that rely on pattern-matching over training data typically do not.

The announcement noted that the public state-of-the-art score had moved from 20% when the benchmark was introduced to 34% by mid-2024, still well short of typical human performance on the same task set. Chollet and Knoop framed the competition as a response to what they described as stagnation on the problem: despite several years of rapid progress on other benchmarks, ARC-AGI scores had improved only slowly, which they took as evidence that leading language models were not developing the kind of general, sample-efficient reasoning the benchmark was designed to test. The prize required winning approaches to be published openly, aiming to keep any advances shared across the field rather than captured by a single lab.

ARC Prize became an annual competition and a recurring reference point in arguments over what benchmark saturation elsewhere in the field actually demonstrated: models could set state-of-the-art results on professional exams and coding tests while still scoring far below humans on ARC-AGI’s puzzle-like reasoning tasks, a gap Chollet and Knoop cited as evidence that scaling existing architectures was not, on its own, closing the distance to general intelligence.