LiveCodeBench

By early 2024, two of coding’s most-cited benchmarks had a shared problem: HumanEval and MBPP had been sitting on the open web for years, and nobody could rule out that their exact problems and solutions had been absorbed into the training data of the models being scored against them. LiveCodeBench, published by researchers at UC Berkeley, MIT and Cornell, answered with a benchmark that could not go stale in the same way: it continuously pulls fresh problems from LeetCode, AtCoder and Codeforces contests and tags each with its publication date, so a model’s score can be split between problems that predate its training cutoff and ones that came after.

The date-segmented design did what it was built to do — it produced direct evidence that some models scored measurably better on older, plausibly-memorised problems than on newer ones, turning contamination from a suspicion into something a benchmark could actually show. Beyond plain code generation, LiveCodeBench also grades self-repair, test-output prediction and code execution, giving a fuller picture of coding competence than a single pass/fail generation task. It was adopted quickly and is now commonly reported across major labs’ model cards — DeepSeek, Google and Alibaba’s Qwen all cite it, though Anthropic and OpenAI tend to lead their own headline tables with SWE-bench instead.

That refreshing pool is also what makes LiveCodeBench numbers easy to misread: because the benchmark fixes a May-2023 start and only extends the end date with each version (v6 reaches April 2025), two “LiveCodeBench” scores are comparable only if they cover the same window — the same Gemini 2.5 Pro build scored 75.6% on one window and 69.0% on a later one. On the benchmark’s own official cross-lab leaderboard, for the August-2024–May-2025 window, OpenAI’s o4-mini led at 80.2%, ahead of DeepSeek’s R1-0528 at 73.1% and Gemini 2.5 Pro at 73.6% — a reminder that it is a genuinely multi-lab board, not a DeepSeek showcase. That official board has not since added the late-2026 frontier; independent third-party runs put the current top models (Claude Fable 5, Claude Opus 5, Gemini 3.1 Pro) near 90% on v6.

Because the problem pool keeps refreshing, LiveCodeBench has so far avoided the saturation that overtook its predecessors, and its live-update approach became a template other domains borrowed when they ran into the same contamination problem. It is not the final word on coding ability — the underlying problems are still competitive-programming puzzles rather than the messy, multi-file work of real software engineering, which is closer to what benchmarks like SWE-bench try to test — but as a memorisation-resistant coding number, it has become a common citation.