Researchers demonstrate practical model-stealing attack against production LLM APIs

A team of researchers led by Nicholas Carlini, with collaborators at Google DeepMind and other institutions, published an attack that extracted precise, nontrivial internal information from commercial language models exposed only through an API — the first such extraction from black-box production systems, according to the paper.

The attack targeted the final embedding projection layer, the component that maps a model’s internal hidden state to output token probabilities. By querying the API and analysing the structure of the returned logits, the researchers recovered the complete projection matrices of OpenAI’s ada and babbage models for under $20 in API calls, confirming their hidden dimensions were 1,024 and 2,048 respectively — figures OpenAI had not published. Applying the same method to gpt-3.5-turbo, they determined its hidden dimension and estimated that extracting its full projection matrix would cost under $2,000 in queries; they also recovered comparable information from Google’s PaLM-2.

The paper argued the technique demonstrated that “black-box” API access does not fully protect a model’s architecture from a sufficiently determined and well-resourced attacker, and it discussed possible defences, including added noise or restricted API outputs. OpenAI and Google were notified before publication and both made changes to mitigate the specific attack described. The result did not expose model weights or training data, but it established that architectural details long treated as trade secrets could leak through ordinary paid API access, adding a new category to the security concerns around closed commercial models.