CORE-Bench

CORE-Bench asks a question that sits underneath most AI-for-science claims: can a model actually run someone else’s research and get the same answer? Built by researchers at Princeton, it takes 270 tasks from 90 published papers in computer science, social science and medicine, each drawn from the Code Ocean platform where authors package their code and data for reproducibility. An agent is handed a paper’s repository and a question about one of its results, and can only answer correctly by getting the original analysis to actually run.

The tasks come in three tiers of difficulty. In the easiest, an agent can answer from outputs the authors already generated; in the middle tier it must run the paper’s code inside a Docker environment the authors provided; in the hardest, it gets only a raw, unconfigured codebase and has to make it work itself — closer to what a researcher attempting to build on someone else’s work actually faces. In the 2024 launch paper, a specialised CORE-Agent built on GPT-4o solved 45.9% of tasks overall but only 21% of the hardest tier, and a more generic AutoGPT-style agent on the same underlying model did substantially worse.

The authors framed the benchmark around scientific credibility rather than raw capability: an agent that can reliably reproduce existing computational results is a precondition, they argued, for one that could eventually be trusted to conduct or check novel research. Because reproduction failures in science are common and costly by themselves, CORE-Bench’s low scores at launch reflect a real and unresolved difficulty rather than a benchmark artefact, and it has not shown the rapid saturation seen on more heavily contested coding and knowledge benchmarks.