vLLM releases PagedAttention inference engine

Researchers at UC Berkeley released vLLM, an open-source library for serving large language models, built around a memory-management technique called PagedAttention. The system had already been running in production behind LMSYS’s Chatbot Arena and Vicuna demo for roughly two months before the public write-up.

The problem it addressed was mundane but expensive: serving an LLM requires storing a growing cache of key-value attention states for every active request, and existing systems allocated that cache in large contiguous blocks sized for the worst case, wasting a reported 60–80% of GPU memory to fragmentation and over-reservation. PagedAttention borrowed the idea of paged virtual memory from operating-system design, partitioning the cache into small fixed-size blocks that could sit anywhere in memory and be shared between requests — for example across the multiple candidate completions of a single prompt. The project reported this cut memory waste to under 4% and, in its own benchmarks, delivered up to 24 times the throughput of Hugging Face’s Transformers library and up to 3.5 times that of Hugging Face’s Text Generation Inference server on equivalent hardware.

Because it was released as open-source software rather than a hosted service, vLLM was adopted directly by other projects and companies building on open-weight models, for whom inference cost — not training cost — was often the binding constraint on what they could ship. It became one of the standard serving engines in the open-model ecosystem, alongside alternatives such as Text Generation Inference and, later, SGLang, and its authors’ subsequent papers on the technique were widely cited in follow-on inference research.

The release is a reminder that a meaningful share of the progress attributed to “better models” in this period was, in fact, better plumbing: the same weights served several times more cheaply.