Microsoft researchers publish LoRA

Microsoft researchers led by Edward Hu published “LoRA: Low-Rank Adaptation of Large Language Models,” proposing a way to adapt a large pretrained model to a new task without updating most of its weights. The method freezes the original parameters and instead injects small, trainable low-rank matrices into each layer of the transformer, learning only those matrices during fine-tuning while leaving the base model untouched.

Against full fine-tuning of GPT-3 175B with the Adam optimiser, the authors reported cutting the number of trainable parameters by a factor of 10,000 and GPU memory requirements by a factor of three, while matching or slightly exceeding full fine-tuning’s performance on several benchmarks across models including RoBERTa, DeBERTa, GPT-2 and GPT-3. Because the low-rank matrices can be merged back into the frozen weights after training, LoRA added no extra latency at inference time — unlike earlier adapter-based approaches, which left a permanent computational overhead.

The practical consequence was that fine-tuning a model the size of GPT-3 no longer required the memory and storage budget of a full duplicate copy of the model for every downstream task. A single frozen base model could support many small, swappable sets of adapted weights, each a fraction of the size of the model itself.

LoRA’s significance became fully visible only once open-weight models proliferated the following year: it became close to the default technique for the hobbyist and research community fine-tuning Stable Diffusion checkpoints and open language models on consumer graphics cards, and variants of the method (QLoRA and others) extended it further as models grew larger. What began as a Microsoft efficiency paper became one of the pieces of infrastructure the open-weight ecosystem was built on.