ARC Prize publishes HRM analysis

The Hierarchical Reasoning Model, a 27-million-parameter architecture with two coupled modules for planning and execution, had drawn wide attention after its creators reported it reached 41% on ARC-AGI-1 using only 1,000 training tasks — a striking result for a model orders of magnitude smaller than the large language models more commonly used on the benchmark. The ARC Prize Foundation, which runs the ARC-AGI benchmark, published an independent analysis testing where that performance actually came from.

Verifying the model against ARC Prize’s own held-out, semi-private test set produced lower scores than the original claim — 32% on ARC-AGI-1 and 2% on the newer, harder ARC-AGI-2 — and a series of ablations found the model’s distinctive hierarchical, brain-inspired architecture was not the main driver of its performance. Replacing the planner-worker split with a same-sized standard transformer produced results within a few percentage points; the largest single gain instead came from letting the model iteratively refine its answer across multiple passes, with a further share coming from data augmentation — generating rotated and flipped variants of training tasks — where a few hundred variants captured most of the benefit the original paper reported from a thousand.

The analysis was a case study in a recurring benchmark-era problem: a headline result attributed to a specific architectural innovation turned out to depend mainly on generic training tricks that could be added to conventional architectures. It reset the interpretation of the result, from an architectural breakthrough in small-model reasoning to a demonstration that certain training techniques transfer well to ARC-AGI regardless of network design.