HARP
also: Human Annotated Reasoning Problems
Whether a model's maths-competition accuracy holds up as problems get harder, using six difficulty tiers built from seven decades of US national competitions.
Albert S. Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse and Aaditya K. SinghReleased 11 December 2024Live
By late 2024, the MATH dataset that had anchored maths-reasoning evaluation for several years was running out of headroom — o1-mini scored 90.0% on it, Gemini 1.5 Pro 86.5%, leaving little room to distinguish one frontier model from the next. HARP was built to reopen that gap. Albert Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse and Aaditya Singh assembled 5,409 problems from seven decades of US national competitions — AHSME, AMC, AIME and USA(J)MO, spanning 1950 to September 2024 — and sorted them into six difficulty tiers calibrated to how each contest’s own field of human competitors had actually performed.
The design worked as intended. Against the same models’ near-ceiling MATH scores, HARP’s hardest 197-problem bracket dropped o1-mini to 41.1% and Gemini 1.5 Pro to 9.6% — evidence that strong performance on an easier competition-maths set did not transfer cleanly to harder problems from the same competitions. Most of the dataset (4,780 problems) keeps a checkable short answer, verified with SymPy, and a 310-problem subset requires full proofs, letting the same collection support both automatic and human grading.
HARP has not become a routine line in model release announcements the way AIME and HMMT have, and its release notes do not describe a live leaderboard tracking newer models against it. Its dataset is also frozen at problems through September 2024, which leaves it exposed to the same contamination risk that live platforms such as MathArena were built to avoid — a fixed competition archive, however large, ages the same way AIME’s did once enough model training runs have had the chance to see it.
The set
5,409 problems drawn from AHSME/AJHSME, AMC, AIME and USA(J)MO contests held between 1950 and September 2024; 4,780 have short answers checkable automatically (verified with SymPy), 4,110 of those also carry multiple-choice options, and a further 310 are proof-based. Problems are grouped into six difficulty tiers calibrated to each contest's historical results, with two independent human-written solutions per problem.
Example
Level-6 (hardest tier) item, AIME 1983 Problem 13: For {1, 2, 3, ..., n} and each of its non-empty subsets a unique alternating sum is defined as follows. Arrange the numbers in the subset in decreasing order and then, beginning with the largest, alternately add and subtract successive numbers. For example, the alternating sum for {1, 2, 3, 6, 9} is 9-6+3-2+1=5 and for {5} it is simply 5. Find the sum of all such alternating sums for n=7. (Answer: 448)raw.githubusercontent.com
Where it stands
The easier tiers are close to saturated for frontier models, but the hardest 197-problem bracket stayed under 50% accuracy for the best model tested at launch. The dataset stops at problems through September 2024 and has not been publicly refreshed since, so later competitions and later frontier models are not covered without an update — and, sourced as it is from public competition archives, it carries the same contamination risk as AIME and HMMT as time passes.
How the top score changed hands
- December 2024Gemini 1.5 Pro9.6%Hardest 197-problem bracket, against 86.5% for the same model on the easier, near-saturated MATH benchmark.
- December 2024o1-mini41.1%Hardest 197-problem bracket, against 90.0% for the same model on MATH.
Current best: o1-mini — 41.1% Best-reported accuracy on HARP's hardest 197-problem bracket at the paper's release; Gemini 1.5 Pro scored 9.6% on the same bracket. No later frontier-model evaluation on HARP has been publicly reported.