MGSM

MGSM tests something narrower than general multilingual knowledge: can a model do the same step-by-step arithmetic reasoning in Swahili or Telugu that it can in English? Researchers at Google built it by manually translating 250 problems from the English grade-school math set GSM8K into 10 typologically diverse languages, so the underlying reasoning task stays fixed while only the language changes.

The original paper’s headline finding was that chain-of-thought reasoning transferred across languages more readily than expected, even into languages with very little representation in training data — PaLM-540B reached 55.0% average accuracy in its best configuration, and scored over 40% on Swahili and Bengali despite each making up under 0.01% of its training corpus. That result helped establish that large models’ reasoning ability was not simply an artefact of English-language training, and MGSM became a standard line in frontier model reports alongside general-knowledge benchmarks like MMLU.

MGSM’s limitation is the one built into its design: because the underlying problems started in English and were translated outward, a model that reasons well only after implicitly translating into English can still score well. OpenAI’s IndQA, released three years later, was explicitly framed as testing knowledge that depends on cultural context rather than translated general reasoning — a distinction its authors drew directly against MGSM and MMMLU.