Hugging Face releases StarCoder

BigCode, a collaboration jointly led by Hugging Face and ServiceNow, released StarCoder, a 15-billion-parameter model for code generation trained on roughly 1 trillion tokens drawn from permissively licensed GitHub repositories spanning more than 80 programming languages, along with Git commits, issue discussions and Jupyter notebooks. The project released both a base model, StarCoderBase, and a variant further fine-tuned on Python.

The model supported a context length of over 8,000 tokens, which BigCode said exceeded any other openly available language model at the time. On HumanEval, a standard code-generation benchmark, StarCoder was reported to score 40.8% with optimised prompting, ahead of OpenAI’s code-cushman-001, and 52.7% on the MBPP benchmark, also ahead of the closed alternatives it was compared against.

Weights were released under BigCode OpenRAIL-M, a version of the Responsible AI Licence designed to permit commercial use while attaching behavioural-use restrictions — a middle position between fully permissive licences such as Apache 2.0 and closed, API-only access. The release extended the open-model wave of spring 2023, alongside efforts such as MosaicML’s MPT-7B days later, into code generation specifically, an area where OpenAI’s Codex and GitHub Copilot had until then had no openly weighted competitor of comparable scale.