Google announces PaLM at 540 billion parameters

Google announced PaLM (Pathways Language Model), a 540-billion-parameter dense decoder-only transformer trained using Pathways, a new distributed-computation system built to orchestrate training across multiple accelerator pods rather than a single one. Google described it as the first large-scale use of Pathways, scaling training to 6,144 TPU v4 chips split across two Cloud TPU v4 pods, and reported 57.8% hardware FLOPs utilisation — which it called the highest yet achieved for a language model at that scale.

Google reported state-of-the-art few-shot performance on the large majority of the roughly two dozen English natural-language-processing tasks it evaluated, beating prior models by wide margins on many of them, and argued that PaLM’s results showed performance gains from scale had not yet plateaued — evidence, at the time, for continuing to build larger models rather than a case that returns were diminishing. The blog post’s most widely circulated illustration was qualitative rather than a benchmark score: PaLM could produce plausible explanations of novel jokes it had not seen before, which Google presented as evidence of multi-step logical inference combined with world knowledge rather than memorised pattern-matching.

PaLM became the base model behind the chain-of-thought prompting result published the same week, where prompting it with worked examples reached state-of-the-art accuracy on a grade-school math benchmark. Along with GPT-3 and the compute build-out that hardware like NVIDIA’s H100 was enabling, PaLM was one of the clearest public data points for the argument that scaling parameter count and compute together, without new architectural ideas, kept producing better models — a premise later tested and complicated by more efficient, differently trained systems.