RE-Bench

RE-Bench measures a capability that safety researchers care about specifically: not whether a model can write code, but whether an AI agent can do the open-ended research-engineering work that could, in principle, let AI systems improve AI systems. METR, a nonprofit built to evaluate frontier models for dangerous capabilities, designed seven ML research-engineering tasks with no single correct answer — optimising a GPU kernel, recovering a model’s performance after its embeddings were corrupted, predicting scaling-law hyperparameters from a handful of experiments — and scored them on a continuous scale rather than pass or fail.

What makes RE-Bench unusual is its human baseline: 61 ML experts ran 71 eight-hour sessions on the same tasks, at the same effective budgets, that the agents were tested on. The comparison turned out to depend heavily on time. Given only two hours, agents built on Claude 3.5 Sonnet and o1-preview beat the human average and iterated more than ten times faster; given eight hours humans had caught up, and by 32 hours the best humans scored roughly double the leading agents, because agents tended to stall on their early approach rather than adapt the way a human researcher would over a long session.

That budget-dependent crossover — AI ahead on short, cheap tasks, humans ahead once sustained judgement matters — became one of RE-Bench’s most-cited findings, feeding directly into METR’s later “time horizon” work measuring how the length of task an AI agent can reliably complete has grown over time. RE-Bench sits alongside MLE-bench and PaperBench as part of a small cluster of evaluations built specifically to track how close AI systems are to automating parts of AI research itself.