DeepSeek releases DeepSeek LLM 67B
DeepSeek's first general-purpose open-weight LLM family, 7B and 67B, trained on 2 trillion English/Chinese tokens.
- Open weights & ecosystem
- Models & capabilities
- Minor
DeepSeek, then a little-known lab spun out of the Chinese quant fund High-Flyer, released DeepSeek LLM, its first general-purpose foundation model family: base and chat variants at 7B and 67B parameters, trained from scratch on a claimed 2 trillion tokens of English and Chinese text. The 7B model used standard multi-head attention; the 67B used grouped-query attention to keep inference cheaper at that size.
DeepSeek published benchmark results showing the 67B base model ahead of Meta’s Llama 2-70B on reasoning, coding, mathematics and Chinese-language comprehension despite having fewer parameters, and reported the chat variant reached 73.78% pass@1 on HumanEval and 84.1% on GSM8K without task-specific fine-tuning — figures the company said also surpassed GPT-3.5 on Chinese benchmarks. The code was released under the MIT licence and the model weights under a separate licence permitting commercial use without a separate application.
The release drew little attention outside specialist coverage at the time — DeepSeek was one of several Chinese labs, alongside 01.AI, shipping competitive open-weight models that autumn. In retrospect it marked the start of a release cadence that would make DeepSeek one of the most closely watched open-weight developers within two years.