Stanford's Alpaca fine-tunes LLaMA for a few hundred dollars

Stanford’s Center for Research on Foundation Models released Alpaca, a 7-billion-parameter model built by fine-tuning Meta’s LLaMA on 52,000 instruction-following examples generated using OpenAI’s text-davinci-003 via the “self-instruct” method. The researchers reported that generating the training data cost under $500 through the OpenAI API, and that fine-tuning took about three hours on eight 80GB A100 GPUs, for a further cost of under $100 — putting the total budget for reproducing ChatGPT-like instruction-following behaviour at roughly $600.

The result behaved qualitatively similarly to OpenAI’s text-davinci-003 on many everyday prompts, according to the researchers’ own comparisons, despite starting from a much smaller base model. Because LLaMA itself had leaked publicly via a torrent posted to 4chan in early March, days after Meta began granting research access, Alpaca’s release demonstrated something more consequential than the model itself: that the gap between a frontier lab’s proprietary chat model and an amateur, low-budget reproduction could be closed in days using data generated by the frontier model itself.

Stanford took the public demo offline about a week after release, citing hosting costs and the limitations of the content filters guarding against unsafe generations — a decision that itself signalled the tension between wide dissemination and responsible deployment that would recur across the open-weight ecosystem.

Because LLaMA carried a non-commercial research licence and OpenAI’s terms of use prohibited using its outputs to train competing models, Alpaca’s method sat on uncertain legal footing on two fronts, a template that dozens of subsequent instruction-tuned “leaked-LLaMA” derivatives copied through 2023 regardless.