FrontierPhysics benchmark finds AI agents fall short on physics research

A large author team listing BenchFlow, a number of universities and some independent researchers posted FrontierPhysics, a benchmark that asks AI agents to carry out physics research from start to finish. It contains 56 tasks written by 42 physicists, each based on the writer’s own research and spread over nine domains of theoretical, computational and experimental physics. The authors report that across 22 models in 26 configurations, “no configuration passes more than 46 of its 168 rollouts”.

Each task requires an agent to review the literature, plan the research, run the work and experiments, and produce structured outputs and a manuscript. Grading combines deterministic verifiers with expert-written rubrics, scored by a reviewer agent, and a rollout passes only if every verifier test and every blocking rubric criterion passes. The paper counts 4,364 rollouts in all, three per task per configuration. The authors say they designed it because existing physics benchmarks test fixed questions, isolated sub-tasks, or reproduction of published papers where the solution is given.

The best result was Claude Opus 5.5 from Anthropic, with a pass rate of 27.4%, which is the 46 of 168 rollouts. Every agent tested was below 30%. The paper notes that pass rate generally rose with cost, although for some Claude models it fell as cost rose, and that OpenAI’s GPT models were generally more cost-efficient. The authors reviewed the trajectories of the 168 GPT-6 Astra rollouts by hand. They concluded that frontier agents’ main weaknesses were “insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors”. With only a few tasks per domain, the paper says differences between domains are uncertain, and it separately assesses how reliable the verifiers and rubrics are, since defects can survive review.

The result sits awkwardly beside a benchmark published a month earlier, on which Astra solved 49 of 50 unpublished theoretical-physics problems. The two are not like for like: that set used closed-ended problems with checkable answers that its authors called “turn-the-crank” research tasks, whereas FrontierPhysics grades whole research outputs on correctness and publishability. The two papers measure different things, so they are not in direct contradiction; together they show that “doing physics” can look close to solved or far from it depending on how the task is set and graded. Neither benchmark has been independently replicated, and each comes from authors rather than from the developers of the models tested.