RealWorldQA

RealWorldQA is unusual among multimodal benchmarks in how little apparatus surrounds it: xAI released it as a plain dataset alongside its Grok-1.5 Vision preview in April 2024, with no accompanying paper and no public leaderboard. Its questions come from ordinary real-world photographs — many taken from inside or around a vehicle — paired with a single verifiable answer, testing spatial reasoning and physical common sense rather than the curated, often text-heavy imagery that dominates academic multimodal benchmarks. The initial release held just over 700 images, and xAI said it intended to grow the set over time.

Because there is no formal leaderboard, RealWorldQA’s numbers travel the way informal benchmarks often do in this record: cited by whichever lab wants to make a comparative claim, in whatever table its own release material assembles. The clearest multi-model comparison available is a third-party one, in Alibaba’s Qwen2.5-VL technical report from February 2025, which put InternVL2.5-78B and the benchmark’s previous open-source state of the art tied at 78.7%, just ahead of Qwen2-VL-72B (77.8%) and GPT-4o (75.4%). Claude 3.5 Sonnet trailed noticeably on the same table, at 60.1% — a large enough gap to suggest the benchmark’s vehicle-and-scene imagery favours some training distributions over others.

RealWorldQA’s appeal is precisely its informality: it asks whether a model’s visual competence generalises to the kind of photograph nobody curated for a benchmark, which is also why it resists the kind of authoritative, single-source leaderboard that benchmarks with a dedicated maintaining team can offer. It continues to appear as a comparison point in vision-language model release material, but — unlike MMMU or ChartQA — with no canonical current leader that this entry could confirm.