OpenAI publishes Codex and the HumanEval benchmark

OpenAI published “Evaluating Large Language Models Trained on Code,” introducing Codex, a GPT model fine-tuned on publicly available GitHub code, and HumanEval, a new benchmark of 164 hand-written programming problems used to measure it. The paper stated plainly that “a distinct production version of Codex powers GitHub Copilot,” which had entered technical preview a week earlier — this paper was the research disclosure behind a product already shipping.

HumanEval departed from prior code-generation benchmarks, which typically scored output by text similarity to a reference solution. It instead measured functional correctness: whether generated code, given a docstring describing the task, actually passed a set of unit tests. On a single attempt per problem, Codex solved 28.8% of HumanEval’s Python problems, against 0% for the unmodified GPT-3 model it was fine-tuned from and 11.4% for EleutherAI’s open GPT-J. Allowing the model 100 samples per problem and picking any that passed pushed the solve rate to 70.2%, a result OpenAI called “a surprisingly effective strategy” and one that shaped how later code models were evaluated and used in practice, with sampling-and-filtering becoming a standard technique.

The paper also documented Codex’s failure modes: it struggled with long, multi-step docstrings and with correctly binding variables across a program, and OpenAI’s accompanying discussion raised concerns about generated code containing insecure patterns and about the effect of automated code generation on software engineering as an occupation.

HumanEval became the reference benchmark for program synthesis for several years, cited by essentially every subsequent code-generating model, until high scores against its relatively small, fixed problem set led researchers to treat it as increasingly saturated and to develop harder successors.