Benchmarks · Coding & software engineering
BigCodeBench
Can a model write a correct program that composes multiple real library functions correctly to follow a complex, multi-step instruction — the kind of task a developer actually does, rather than an isolated algorithm puzzle?
BigCode Project (Hugging Face, ServiceNow & collaborators)Released 22 June 2024Live
BigCodeBench pushes past the isolated-function format that made HumanEval and MBPP easy to saturate. Instead of a single self-contained problem, each of its 1,140 tasks asks a model to write a program that correctly calls on real functions from 139 popular libraries — spanning data analysis, web development and five other domains — to satisfy a compound, multi-step instruction. The authors, working under the open BigCode collaboration between Hugging Face, ServiceNow and academic partners, checked each task with an average of 5.6 automated tests reaching close to full branch coverage, and offered two variants: one giving the model a complete docstring, the other a terse instruction closer to how a developer would actually phrase a request.
The gap the benchmark was built to expose showed up immediately. At publication in mid-2024, the paper reported that models solved “up to 60%” of tasks against a human baseline of about 97% — a wide margin that made clear composing several real function calls correctly was harder than writing one clean algorithm from scratch, even for models that were, by then, comfortably clearing 80–90% on HumanEval. Alibaba’s Qwen2.5-Coder release later that year cited BigCodeBench alongside LiveCodeBench as one of its comparison points, part of a broader shift in coding evaluation toward multi-step, tool-using tasks rather than single-function generation.
BigCodeBench was accepted as an oral presentation at ICLR 2025 and continues to be maintained with an active leaderboard, positioning it as one of several benchmarks — alongside LiveCodeBench and SWE-bench — built specifically to keep measuring real gaps in coding ability after the first generation of code benchmarks stopped being able to.
The set
1,140 tasks requiring calls into 139 real libraries across seven domains (data analysis, web development and others), each checked by an average of 5.6 test cases with roughly 99% branch coverage. A 'Complete' split gives the model a full docstring; an 'Instruct' split rewrites it as a terse natural-language instruction, closer to how a person would actually ask.
Example
One task's docstring, verbatim: 'Calculates the average of the sums of absolute differences between each pair of consecutive numbers for all permutations of a given list. Each permutation is shuffled before calculating the differences.' The model must write a task_func(numbers) implementing this using itertools and random.shuffle.huggingface.co
Where it stands
At publication the paper reported that models scored 'up to 60%' against roughly 97% human accuracy, a wide gap by design; the benchmark is maintained with an active leaderboard and was accepted as an ICLR 2025 oral, and it continues to be cited alongside LiveCodeBench as a harder alternative to HumanEval and MBPP.