Benchmarks · Coding & software engineering
NanoGPT Speedrun
also: modded-nanogpt, Automated LLM Speedrunning Benchmark
Originally, how fast a human team can train a small GPT-2-scale model to a fixed validation-loss target; adapted into an AI-agent benchmark testing whether a model can reproduce a known training-speed improvement itself, given only a hint of what changed.
Keller Jordan & open community; formalised into an agent benchmark by Meta FAIR researchersReleased 28 May 2024Live
The NanoGPT speedrun began as a human competition, not an AI benchmark: starting from Andrej Karpathy’s llm.c replication of GPT-2, an open community led by Keller Jordan set out to train a small language model to a fixed validation-loss target on a fixed set of hardware as fast as possible, with the training data pipeline held constant but everything else — architecture, optimiser, low-level system code — fair game. Records are logged in public, and the chase produced a long series of small, real optimisations, from the Muon optimiser to FlashAttention 3 to FP8 quantisation, that cut the original 45-minute baseline down to a little over a minute within two years.
That record history turned out to be useful for a different purpose. In 2025, researchers formalised it into the Automated LLM Speedrunning Benchmark, which asks whether a reasoning model can reproduce one of the 19 documented historical improvements itself — starting from the code of the record it superseded, and given only a natural-language hint of what changed, rather than the change itself. The question is narrower than open-ended research capability: not whether a model can invent a novel optimisation, but whether it can reliably reimplement one that is already known to work. The paper’s authors reported that current systems struggled with a meaningful share of the 19 tasks even with detailed hints.
Because the underlying speedrun is still an active human competition, the benchmark has a moving target baked into it: every new community record becomes a fresh candidate task for the agent version, so the standard the models are chasing keeps advancing at the same time as the models themselves. That combination — a real, ongoing research competition repurposed as a reproducibility test — makes it an unusually direct measure of whether a model can do incremental ML-systems research rather than just describe it.
The set
The underlying community challenge: train a 124M-parameter model on FineWeb to a validation loss of 3.28 or below on 8 NVIDIA H100 GPUs as fast as possible, with the data pipeline fixed but architecture, optimiser and system code free to change; records are logged publicly and have fallen from 45 minutes in May 2024 to just over a minute by mid-2026. The agent version, published as a formal benchmark in 2025, draws 19 discrete historical improvements from that record chain and tests whether a reasoning model can reimplement each one from the prior record's code, under varying levels of hint.
Example
Excerpt of the Level 2 (text-description) hint for Record 1, verbatim: 'Here's a detailed breakdown of the improvements: 1. Architectural Improvements — Rotary Positional Embeddings: Replaced standard positional embeddings with rotary embeddings ... Benefits: Better captures relative positions and attention patterns, improves model accuracy ... 2. Optimization Improvements — Learning Rate Changes: Increased base LR from 0.0015 to 0.0018 ... Benefits: ... allows more stable high-LR training.' The agent starts from the prior record's training script and is judged on the validation loss and wall-clock time it actually achieves.arxiv.org
Where it stands
The community speedrun remains active, with the record standing at 1.23 minutes as of July 2026 (up from a 45-minute baseline); the formal agent benchmark reported in 2025 that current reasoning LLMs struggled to reproduce several of the 19 historical improvements even with detailed hints.