Timeline

Epoch finds Fable 5.2 and GPT-6.2 Astra short of employee-level performance on real work

Fable 5.2 scored 98% on AI data-centre analysis, which Epoch rates contractor level, against 65% for Astra, which covered one building where Epoch's human output covered six.

  • Benchmarks & progress
  • Notable

Epoch AI, the research group that tracks AI progress, published an evaluation of Claude Fable 5.2 and GPT-6.2 Astra on its own work tasks. It says the two “roughly tie for a new record” on its Capabilities Index, but that strong benchmark results “do not always translate” to real workplace tasks. Epoch ran the two models and four others on an updated version of its Epoch Automation Reports, tasks derived from its own work, graded by hand against internal standards, with 100% meaning the standard it expects of its employees.

Its first finding is that no model met that standard on any task. GPT-6.2 Astra averaged 65%, 17 points below Fable 5.2 despite the near-tie on the index. The widest gap was in AI data-centre analysis, where models add a facility to Epoch’s database using news, regulatory documents and satellite imagery. Anthropic’s Fable 5.2 scored 97.5% and Claude Opus 5.5 87.5%, which Epoch describes as the level of one of its contractors; GPT-6.2 Astra scored 65%. Epoch said the contractor typically spends about 16 hours per data centre, Opus 5.5 took just under 3 hours at lower cost, and Fable 5.2 about 6.5 hours at higher cost, and that it is considering moving contractors’ time toward reviewing AI output. Astra, Epoch said, covered one building although its sources pointed to several, because it would not estimate without a separate source for each number.

On open-ended research tasks Epoch described shortcomings across all models: they reused existing ideas, relied on first attempts, and struggled to identify and present key findings. In a task on out-of-distribution generalisation using chess variants, every model used variants already known to models; Epoch said none was genuinely novel. When asked to compare their own output with a human’s, all models preferred their own in every case but one, where Gemini 4 Argon chose the human’s.

Epoch listed its limits. Each model ran once per task on a limited suite, so scores are “noisy”; grading is subjective, and Epoch says the qualitative findings are more informative than the numbers; and because it changed its setup, results are only roughly comparable with its earlier report. The tasks are Epoch’s own, and Epoch is the grader, which suggests caution in generalising from them.

The piece is independent of both developers. It followed Anthropic’s launch of Fable 5.2 by eight days and came the same day as OpenAI’s release of GPT-6.2 Astra, and added hand-graded evidence on real tasks to the launch benchmarks, as an earlier Epoch test of AI research automation had.

In the commentary

What people were saying around this time — external links, from the record's commentary rail.