Google trains a 1.6-trillion-parameter Switch Transformer
Routing each input to a single expert rather than blending several simplified prior mixture-of-experts designs, and the paper reported up to 7x faster pre-training than a dense T5 baseline.
Models & capabilities