Video-MME

Before Video-MME, most benchmarks billed as testing “video understanding” for multimodal models could largely be solved by looking at one frame. Researchers led by Nanjing University, with collaborators across several Chinese universities, built Video-MME specifically to close that gap: 900 videos totalling 254 hours, spanning everything from short clips to hour-long footage across six visual domains, paired with 2,700 questions that a still image cannot answer. Videos are grouped into short, medium and long buckets, and models can be tested with or without access to the clip’s subtitles.

The benchmark’s headline result at launch, in mid-2024, was as much about a weakness as a leader: every model tested, without exception, got worse as the video got longer, evidence that long-horizon temporal understanding remained a genuine unsolved problem rather than a data-scale issue that would resolve itself. Among commercial models, Gemini 1.5 Pro led clearly, at 75.0% without subtitles against GPT-4o’s 71.9% — a gap the paper’s authors attributed partly to Gemini’s native support for very long context windows, well suited to ingesting an hour of video.

That lead held up for some time: a comparison table in Alibaba’s Qwen2.5-VL technical report, published in February 2025, still put Gemini 1.5 Pro ahead with subtitles supplied, at 81.3%, against Qwen2.5-VL-72B’s 79.1% and GPT-4o’s 77.2%. Video-MME has since become one of the standard citations for video capability in frontier model releases, sitting alongside newer, harder successors — including Video-MMMU, which added a knowledge-acquisition angle — as labs continue pushing on the length and complexity of video a model can genuinely follow rather than skim.