Mamba paper proposes selective state-space models as a transformer alternative

Albert Gu and Tri Dao published Mamba, an architecture built on structured state-space models (SSMs) rather than the attention mechanism underlying every dominant large language model since the original transformer paper. The core idea was making the model’s internal state-space parameters functions of the input token itself — “selective” SSMs — so the model could choose what to propagate or discard as it moved along a sequence, a capacity earlier, input-independent SSMs had lacked and that had limited their competitiveness on language tasks.

Attention’s computational cost grows quadratically with sequence length, because every token attends to every other token; Mamba’s, like other SSM architectures, grows linearly, and the paper paired the selective mechanism with a hardware-aware parallel algorithm designed to make the resulting model fast on GPUs despite abandoning the convolutional tricks earlier SSMs had relied on for efficient training. The paper reported that a 3-billion-parameter Mamba model matched the performance of transformers twice its size on language modelling and matched or outperformed similarly sized transformers across several other modalities, while offering roughly five times the inference throughput and maintaining performance on sequences up to a million tokens long — a length at which standard attention becomes prohibitively expensive.

Mamba did not displace the transformer, and the two architectures’ relative merits on very large-scale training runs remained contested. But the paper renewed a research agenda — reducing attention’s quadratic cost without sacrificing its modelling power — that earlier sub-quadratic proposals had struggled to make competitive, and it was followed within a year by hybrid architectures, such as AI21’s Jamba, that combined SSM layers with a smaller number of attention layers to capture benefits of both approaches.