UK AI Security Institute releases optstop to cut evaluation compute
The open-source tool applies adaptive stopping rules to model evaluations, cutting compute by 57 to 97% in testing without changing the resulting score estimates.
- Safety & alignment
- Minor
The UK AI Security Institute released optstop, an open-source tool that decides, case by case, when a model evaluation has collected enough evidence to stop early, rather than running every item in a benchmark to a fixed sample size.
AISI said the problem it addressed had become a real constraint: evaluating frontier models can now require hundreds of millions of tokens per run, and fixed-sample testing keeps spending compute on questions that are already statistically clear while sometimes running out of budget before resolving the genuinely uncertain ones. Using adaptive stopping rules instead, AISI reported that optstop “saved between 57% and 97% of planned runs” across the conditions it tested, without changing the resulting score estimates. The tool integrates with AISI’s own Inspect evaluation framework, requiring only a few extra lines of configuration, and is released alongside an accompanying paper.
Unlike AISI’s evaluation findings on specific models, this was an infrastructure release aimed at other evaluators rather than a judgement on any AI system — a small, practical contribution to making rigorous testing cheaper at a moment when the compute cost of evaluating frontier models had itself become a bottleneck to doing it thoroughly.