Benchmarks · Science & research
ChemBench
How does a model's chemical knowledge and reasoning compare with a trained human chemist's, and does it know the limits of its own answers?
Jablonka Lab, Friedrich Schiller University JenaReleased 1 April 2024Live
ChemBench was built to answer a provocative question directly in its title: “are large language models superhuman chemists?” A team led by researchers at the Jablonka Lab in Jena assembled 2,788 chemistry questions covering knowledge, calculation, reasoning and safety, then ran a smaller 236-question subset, ChemBench-Mini, past 19 expert human chemists so model scores could be compared against a real, matched human baseline rather than an assumed one.
The headline result was striking on its face: OpenAI’s o1 answered around 92% of the ChemBench-Mini questions correctly, against roughly 50% for the best-performing human chemist in the study. But the paper’s more careful finding was about reliability rather than raw accuracy — models were shown to be overconfident, giving high-certainty answers on questions they got wrong, and to fail on some basic tasks that a trained chemist would not miss, which the authors argued matters more for whether a model can be trusted in a lab setting than the average score does.
ChemBench sits alongside benchmarks like LAB-Bench and SciCode in a push to test whether AI systems have the specific, checkable knowledge a working scientist needs rather than general test-taking ability, and its authors argued the “superhuman on average, unreliable on basics” pattern was itself the more important finding — evidence for treating chemistry-capable models as fast but uneven collaborators rather than replacements for expert judgement.
The set
2,788 question–answer pairs spanning chemistry knowledge, reasoning, calculation and safety topics, drawn up by the research team behind the paper 'Are large language models superhuman chemists?'. A 236-question 'ChemBench-Mini' subset was used to run a matched comparison against 19 expert chemists.
Example
An analytical-chemistry question from the appendix: "Which of the following analytical methods is most appropriate for performing a survey analysis of a solid sample containing various metals? A. X-ray fluorescence analysis B. Differential pulse polarography C. Flame-atomic absorption spectroscopy D. Gas chromatography with flame ionization detector E. Hydride generation atomic absorption spectroscopy"arxiv.org
Where it stands
On the human-comparison subset, OpenAI's o1 scored around 92% against roughly 50% for the best of 19 expert chemists tested — but the authors found models remained overconfident and weak on some basic tasks, so the headline gap does not mean models are reliable chemists.