Agents' Last Exam

Agents’ Last Exam asks a narrower question than most model benchmarks: not what an AI system knows, but whether it can actually finish a piece of professional work end to end, in the real software a specialist would use, with a result that can be checked rather than merely judged plausible. UC Berkeley’s RDI centre, working with more than 250 industry experts, built the set around 55 sub-industries — from Adobe After Effects animation to Siemens NX 3D modelling to neuroimaging analysis in FSLeyes — mapped onto the US government’s own occupational taxonomy rather than an academic subject list.

Each task hands an agent an instruction, the necessary input files and the actual application, then grades the output against a reference answer that stays hidden until the run completes, closing off the usual routes to gaming a test set. The initial release covers over 1,000 tasks, with the project stating an intention to grow the pool toward 5,000. Early results are stark by design: across the hardest difficulty tier, the paper reports an average full pass rate under 1% for current agent systems, a gap the authors attribute less to raw model capability than to a shortage of benchmarks that track sustained performance on real, economically valuable workflows rather than short, single-turn questions.

ALE is explicitly framed as a living benchmark rather than a fixed leaderboard snapshot, with tasks added over time and a public leaderboard tracking submissions as they come in. That design — alongside its scale and its direct anchoring to occupational categories — puts it in the same lineage as OpenAI’s GDPval, part of a broader shift in 2025–26 toward benchmarks built from actual professional output rather than exam-style questions.