OpenAI publishes Codex, an LLM trained on code powering GitHub Copilot

OpenAI published Codex, a version of GPT-3 fine-tuned on billions of lines of publicly available source code from GitHub, and the accompanying paper describing it in detail. Given a natural-language docstring describing a task, Codex generated working code to perform it, translating instructions such as “write a function that returns the sum of two numbers” directly into executable Python.

To measure this, OpenAI introduced HumanEval, a benchmark of 164 hand-written programming problems each paired with unit tests, since prior code-generation evaluations tended to check superficial text similarity to a reference solution rather than whether the generated code actually ran correctly. A 12-billion-parameter Codex model solved 28.8% of HumanEval problems on a single attempt, compared with 0% for the base GPT-3 model of similar size and 11.4% for GPT-J, an open alternative available at the time. Sampling 100 candidate solutions per problem and selecting a working one — a strategy only practical because generating code is cheap relative to writing and testing it by hand — raised the solve rate to 70.2%. The paper also reported limitations: Codex struggled with instructions describing long chains of sequential operations and with correctly binding operations to the right variables across a longer piece of code.

A separate, production version of Codex became the model behind GitHub Copilot, an autocomplete tool embedded directly in code editors that GitHub had begun previewing the previous month. Codex and Copilot together established code generation as a commercially viable product category built on a general-purpose language model rather than a specialised system, and HumanEval became a standard reference benchmark that subsequent code-generating models were routinely measured against. Codex is a direct ancestor of the coding agents — including OpenAI’s own Codex CLI — that followed years later.