DeepSeek releases DeepSeek-V2

DeepSeek released DeepSeek-V2, a 236-billion-parameter mixture-of-experts language model of which only 21 billion parameters are active for any given token, with a 128,000-token context window. The company published both the weights and an accompanying technical paper.

The paper introduced two architectural components that later carried through DeepSeek’s subsequent models. Multi-Head Latent Attention compresses the key-value cache used during inference into a much smaller latent vector, which DeepSeek said cut KV cache size by over 90% compared with its earlier 67B model — the main determinant of how much memory is needed to serve long contexts. DeepSeekMoE, its mixture-of-experts design, was intended to make training more economical by routing each token through only a fraction of the model’s total parameters. Taken together, DeepSeek reported a roughly 42% reduction in training cost and a more than five-fold increase in maximum generation throughput relative to its predecessor. The model was pretrained on 8.1 trillion tokens before supervised fine-tuning and reinforcement learning.

DeepSeek reported that despite activating only a fraction of its total parameters per token, V2 reached performance competitive with the strongest open-weight models available at the time. Independent verification of exact benchmark standings was limited in the way typical of self-reported lab papers, but the architecture itself was not disputed and proved durable: both Multi-Head Latent Attention and DeepSeekMoE reappeared, refined, in DeepSeek-V3 and in the R1 reasoning model that followed. V2’s efficiency claims — training a large model cheaply by keeping most of its capacity dormant on any single forward pass — became a reference point in arguments about how much frontier-level capability required frontier-level compute spend, an argument DeepSeek sharpened considerably with its later releases.