QLoRA makes fine-tuning fit on one GPU

Researchers at the University of Washington — Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer — published QLoRA, a method for fine-tuning large language models that they reported cut the memory required enough to fine-tune a 65-billion-parameter model on a single 48GB GPU — hardware within reach of a well-equipped individual researcher rather than requiring a multi-GPU server.

The method combined three techniques: backpropagating gradients through a frozen language model that had been quantised to a new 4-bit format designed for normally distributed weights (“NF4”), a second round of quantisation applied to the quantisation constants themselves to shrink memory further, and paged optimizers to handle memory spikes during training. The paper reported that models fine-tuned this way matched the performance of full 16-bit fine-tuning, a result the authors said was not obvious in advance given how aggressively the base model’s weights had been compressed. Using QLoRA, the authors trained a family of chatbot models called Guanaco and reported that the largest reached 99.3% of ChatGPT’s performance, by their evaluation, after roughly 24 hours of fine-tuning on one GPU.

The paper landed squarely in the open-versus-closed argument that had built through the spring of 2023: where training a frontier-scale model from scratch remained the preserve of well-funded labs, QLoRA meant that adapting an already-trained open-weight model to a new task or persona no longer did. It became one of the most widely adopted fine-tuning techniques in the open-model ecosystem, cited in a large share of subsequent papers and community fine-tunes released on Hugging Face.