Benchmarks · Reasoning & problem-solving
ARC (AI2 Reasoning Challenge)
also: AI2 Reasoning Challenge, ARC-Challenge, ARC-Easy, AI2-ARC
Whether a system can answer grade-school science questions that simple retrieval and word-co-occurrence methods get wrong — a test built specifically to exclude questions that lookup alone could solve.
Allen Institute for AI (AI2)Released 14 March 2018Retired
ARC predates the models this site tracks, but its design shaped every “hard subset” benchmark that followed it, including its unrelated 2024 namesake ARC-AGI. AI2 researchers built the dataset in 2018 from 7,787 real grade-school science questions, then split it deliberately: any question a simple retrieval system or a word-co-occurrence algorithm could already answer went into an Easy set, and everything those methods got wrong became the Challenge set. At launch, even neural models transferred from strong question-answering benchmarks such as SQuAD could not significantly beat random guessing on the Challenge questions.
That gap closed quickly once large language models arrived. By 2020, GPT-3’s 175-billion-parameter model scored 70.1% on ARC-Easy and 51.5% on ARC-Challenge in a few-shot setting — well above the 2018 baselines, though the paper itself noted the gain scaled only shallowly with model size and still trailed the fine-tuned state of the art. ARC nonetheless remained a common secondary citation through the early 2020s, in the same family as HellaSwag and WinoGrande, as a quick check on commonsense and grade-school reasoning ability.
As frontier labs moved to explicitly graduate-level tests such as MMLU and later GPQA, ARC’s grade-school questions stopped being difficult enough to distinguish models from each other and largely dropped out of headline model-release reporting. It persists mainly as one entry in standardised academic evaluation suites rather than as a benchmark any lab still designs a release around.
The set
7,787 genuine grade-school multiple-choice science questions, split into an Easy set (5,197 questions) and a harder Challenge set (2,590 questions) reserved for items that both a retrieval-based algorithm and a word-co-occurrence algorithm answered incorrectly. Released with a companion corpus of 14 million relevant science sentences for open-book approaches.
Example
Which property of a mineral can be determined just by looking at it? (A) luster (B) mass (C) weight (D) hardness (Answer: A)arxiv.org
Where it stands
Neural baselines at launch could not significantly beat random guessing on the Challenge set; within two years GPT-3 was scoring above 70% on the Easy set, and ARC dropped out of frontier model announcements well before 2020 as later benchmarks such as MMLU took over as the standard citation.
How the top score changed hands
- March 2018Neural QA baselines (2018)did not significantly beat random guessing (Challenge set)Included leading neural models transferred from SQuAD and SNLI; none cleared the random baseline on the harder Challenge subset.
- May 2020GPT-3 (175B, few-shot)70.1% (Easy) / 51.5% (Challenge)Reported as 'relatively shallow scaling with model size'; still below the fine-tuned state of the art at the time.