DeepMind publishes Gopher, RETRO and a risk taxonomy together
A 280-billion-parameter model, a smaller retrieval-augmented alternative that matched larger models, and a taxonomy of six categories of language-model harm.
- Models & capabilities
- Safety & alignment
- Notable
DeepMind published three pieces of research together, treating scale, efficiency and harm as one connected question rather than three separate ones. Gopher, a 280-billion-parameter transformer language model, was the company’s answer to GPT-3-scale training: DeepMind reported that it outperformed contemporary large models on reading comprehension, fact-checking and identifying toxic language, while still showing weak performance on tasks requiring logical or mathematical reasoning and common-sense inference — a pattern that fed the developing argument that scale alone did not close every capability gap evenly.
RETRO, a retrieval-augmented transformer, was published as a direct counterpoint. Rather than storing all of its “knowledge” in parameters, RETRO looked up relevant passages from a large text database at inference time and conditioned its output on them, achieving performance comparable to much larger dense models with an order of magnitude fewer parameters. DeepMind framed retrieval as a more efficient and more transparent path to capability than raw scale, since an output could be traced back to the specific passages that informed it.
The third piece, a paper cataloguing the “ethical and social risks of harm from language models,” organised the field’s concerns into six categories: discrimination and exclusion, information hazards, misinformation harms, malicious uses, human–computer interaction harms, and harms from automation and unequal access. The authors argued that existing benchmarks under-measured several of these risk categories, particularly misinformation, and that mitigations for embedded stereotypes remained underdeveloped.
Many of the harms in this document are hard to measure and thus hard to mitigate, in part due to a lack of appropriate datasets, metrics, or the ongoing work required to establish them.
— DeepMind, “Ethical and social risks of harm from Language Models”
Taken together, the release functioned as an early statement that a frontier lab could pursue scale, efficiency and systematic harm assessment as parts of the same research programme rather than trade-offs against one another.