Hugging Face releases StarCoder2
The Stack v2 dataset grew to roughly ten times the size of its predecessor, and BigCode said the 15B model matched benchmarks of models more than twice its size.
- Open weights & ecosystem
- Models & capabilities
- Minor
The BigCode project, a collaboration jointly led by Hugging Face, ServiceNow and NVIDIA, released StarCoder2, a successor to the original StarCoder released the previous year. Three sizes were released — 3 billion, 7 billion and 15 billion parameters — trained respectively by ServiceNow, Hugging Face and NVIDIA, each covering more than 600 programming languages and trained on between roughly 3 and 4 trillion tokens depending on size.
The models were trained on The Stack v2, a new open code dataset built from the Software Heritage archive that BigCode said was roughly ten times larger than the original Stack dataset used for StarCoder, growing from around 2.9 terabytes to more than 32 terabytes of deduplicated source code. BigCode reported that the 15-billion-parameter model matched the performance of code models more than twice its size on standard evaluations, and that the smallest, 3-billion model matched the performance of the original 15-billion-parameter StarCoder — presenting the release chiefly as an efficiency gain rather than a scale increase.
The two smaller models were designed to run on widely available consumer GPUs, extending practical local use of a capable open code model beyond users with data-centre-class hardware. As with the original StarCoder, the release continued BigCode’s stated emphasis on data transparency and governance ahead of training a model on public code, including tooling for code owners to check whether their repositories were included in the training data.