Benchmarks · Science & research
SciCode
Can a model write the code a scientist actually needs — numerical methods, simulations and calculations — to solve a real research problem, built up step by step rather than in one shot?
UIUC, Argonne National Laboratory & University of Chicago (Minyang Tian et al.)Released 18 July 2024Live
SciCode tests a step that most coding benchmarks skip: writing the actual numerical and simulation code a working scientist needs to answer a research question, not a self-contained programming puzzle. A large team of researchers, led by Minyang Tian and Luyu Gao with scientists across sixteen subfields of physics, maths, chemistry, biology and materials science, built 80 problems drawn from real research and broke each one into a sequence of smaller, gradable subproblems — 338 in total — complete with reference solutions and test cases the scientists themselves wrote.
The decomposition matters because research code is rarely one function: a simulation typically needs several supporting pieces built and verified in order before the interesting result appears, and SciCode scores a model’s ability to produce that whole chain correctly. At release, the results showed how far the task still was from solved — Claude 3.5 Sonnet, the best model tested, completed only 4.6% of main problems when given the realistic amount of scientific context, with most other frontier models of the time scoring lower still.
By early 2025, OpenAI’s o3-mini variants had moved the benchmark’s leaderboard to around 9–11%, still a small fraction of the problem set. SciCode’s authors designed it deliberately to resist the fate of benchmarks like MMLU, where models plateau near a ceiling within a year or two of release; the low scores and step-by-step structure were meant to keep headroom for measuring real progress in AI-assisted scientific coding as models improve.
The set
80 challenging main problems, drawn from real research across 16 subfields in physics, maths, chemistry, biology and materials science, decomposed by the scientists who wrote them into 338 gradable subproblems with reference solutions and test cases; each subproblem can carry optional scientific background text.
Example
From the "Berendsen_thermostat" problem's first subproblem, the literal prompt given to the model: "Wrap to periodic boundaries. Implementing a Python function named `wrap`. This function should apply periodic boundary conditions to the coordinates of a particle inside a cubic simulation box." — one of several subproblems that build toward simulating a full molecular-dynamics thermostat and barostat.github.com
Where it stands
At release the best model, Claude 3.5 Sonnet, solved only 4.6% of main problems in the most realistic setting; the GitHub leaderboard showed OpenAI's o3-mini variants leading with around 9–11% by February 2025 — still far from saturated.
How the top score changed hands
Current best: OpenAI o3-mini-low — 10.8% (main problems) Leaderboard figure from the SciCode GitHub repository; date approximate (as of February 2025).