GSM8K

GSM8K asks a narrower question than its reputation suggests: not whether a model knows advanced mathematics, but whether it can chain two to eight elementary arithmetic steps together without losing the thread. OpenAI researcher Karl Cobbe and colleagues built 8,500 grade-school word problems by hand, reasoning that a task simple enough for a child would still expose step-by-step reasoning versus pattern-matching. Grading is strict — only the final number counts — and the paper’s real contribution was showing that training a separate “verifier” to rank candidate solutions beat scaling up the solving model alone.

The benchmark became a standard reasoning citation almost immediately: chain-of-thought prompting used it as a headline demonstration in 2022, showing that simply asking a model to show its working beat OpenAI’s fine-tuned-and-verified baseline with no retraining. By GPT-4’s 2023 arrival scores were above 90%, though OpenAI flagged that some GSM8K training examples had leaked into its pre-training data — an early instance of the contamination problem that would dog the benchmark.

That question was tested directly in 2024, when Scale AI built GSM1k, a fresh problem set matched to GSM8K’s style and difficulty; several open model families dropped by double digits while frontier closed models barely moved. GSM8K is now widely treated as saturated — a 2025 MIT study argued the plateau near 95% partly reflected label noise rather than genuine parity — and many current frontier releases no longer report a GSM8K score at all, having moved to harder tests.