Aider Polyglot

Most coding benchmarks ask a model to write a function from a description. Aider’s polyglot benchmark asks something closer to a real workday: given an existing codebase, can the model make the requested change correctly, in the actual files, in whatever language that project happens to use? Paul Gauthier, the developer of the open-source Aider coding tool, built the benchmark in December 2024 from 225 Exercism exercises across six languages, deliberately chosen because they had beaten most top models at the time — his earlier Python-only benchmark had saturated above 80% and stopped distinguishing anything.

The benchmark runs entirely inside Aider itself, so a model’s score reflects two things at once: whether its proposed fix is actually correct, and whether it can express that fix in the tool’s file-editing format without corrupting the codebase. At launch, OpenAI’s o1 topped the table at 61.7%, which Gauthier called strong but left plenty of headroom. That headroom closed fast: Claude 3.7 Sonnet reached 64.9% two months later, Gemini 2.5 Pro’s June 2025 update reached 83.1%, and by August 2025 GPT-5 running with high reasoning effort scored 88.0% — a jump of nearly 30 points in under a year.

Because Aider is a real tool used by working developers rather than a synthetic harness, and because its leaderboard reports the dollar cost of each run alongside the score, the benchmark doubles as a rough guide to the price-performance trade-off between frontier models — a dimension most academic coding benchmarks leave out entirely. It has become a standard citation in coding-model release announcements, including Anthropic’s Claude Opus 4.5, which reported roughly a 10% improvement over its predecessor on the benchmark without disclosing an absolute score.

Two caveats matter when reading the number. Because the score depends on a model expressing its fix in Aider’s strict diff-edit format, a vendor’s own eval can report a different figure — DeepSeek’s R1 card, for instance, listed a lower Aider score than Aider’s own run produced, a gap Aider attributes to format-adherence penalties its harness applies. And the public board itself has gone quiet: its last additions were in October 2025, so it does not yet rank the 2026 frontier, and GPT-5’s 88.0% stands as the last recorded top rather than a live one.