Scale AI publishes GSM1k contamination study of GSM8K

Scale AI researchers published GSM1k, a new set of roughly a thousand grade-school arithmetic word problems built to match the style, difficulty and human solve rate of GSM8K, the benchmark most language models were reporting scores against. The point was to test whether GSM8K scores reflected genuine mathematical reasoning or memorisation of a widely-circulated public dataset.

The comparison found meaningful, uneven contamination. Several model families — the paper singled out Mistral’s and Microsoft’s Phi models — showed accuracy drops of up to 13 percentage points moving from GSM8K to the new, unseen problem set, with the degradation showing up “systematically” across almost every size in those families rather than as an occasional outlier. The researchers found a positive correlation between how readily a model could reproduce GSM8K examples verbatim and the size of its performance gap on GSM1k, consistent with the drop being memorisation rather than a difference in problem difficulty. Frontier closed models — GPT-4, Gemini and Claude among them — showed comparatively little gap, suggesting their higher GSM8K scores were not simply a training-data artefact.

The study did not accuse any lab of deliberately training on benchmark data; contamination of this kind typically enters through web-scraped pretraining corpora that happen to contain the benchmark itself, or discussions and solutions to it, without a developer’s specific intent. But the finding gave concrete, model-by-model numbers to a suspicion that had been circulating informally for over a year: that headline benchmark scores for open and smaller models were not always comparable to each other, let alone to frontier models, because the test itself was no longer clean. GSM1k became one of the more frequently cited empirical contributions to the broader argument that public, static benchmarks decay in value the longer they are public.