Microsoft announces Turing-NLG at 17 billion parameters

Microsoft Research announced Turing Natural Language Generation (T-NLG), a 78-layer Transformer with 17 billion parameters that it described as the largest language model published to date. The model set new state-of-the-art results on the WikiText-103 and LAMBADA language-modelling benchmarks, beating NVIDIA’s Megatron-LM, and Microsoft demonstrated it performing abstractive question answering and summarisation with little task-specific supervision — going beyond the pattern completion GPT-2 had shown.

Training a model of this size required new infrastructure. Microsoft trained T-NLG on a cluster of NVIDIA DGX-2 hardware connected by InfiniBand, using the ZeRO optimizer inside its newly released DeepSpeed library to fit the model and its optimizer state across GPUs without the memory blow-up that had previously capped model size on comparable hardware.

T-NLG’s claim to being the largest published model lasted only a few months: OpenAI’s 175-billion-parameter GPT-3, described in a paper the following May, made Turing-NLG’s 17 billion look modest by comparison. The entry’s lasting significance is less the model itself than the marker it set of how quickly the parameter count that counted as “large” was moving in early 2020, and DeepSpeed’s ZeRO optimizer went on to become widely used infrastructure for training models beyond a single GPU’s memory.