ERQA

ERQA — Embodied Reasoning QA — tests whether a multimodal model can look at a real-world scene and answer the kind of question an agent would need to answer before acting in it: where objects are relative to one another, which way something is moving, what a scene would look like from another viewpoint, which item to reach for. Google DeepMind’s Gemini Robotics team built it as a 400-question, multiple-choice test and open-sourced it alongside the March 2025 Gemini Robotics release, drawing its images from a mix of researcher photographs and existing robotics datasets. About a quarter of the questions span more than one image, forcing the model to hold a scene together across views rather than reason from a single frame.

Because answers are single letters checked by exact match, ERQA grades cleanly at scale, which is part of why the Center for AI Safety’s AI Dashboard adopted it as one of its standardized third-party evaluations — an independent run across frontier models rather than a set of vendor self-reports. On that dashboard the progression is a steady climb: multimodal models sat below 50% in 2024, reasoning models such as o3 pushed past 60% by the end of that year, and Google’s own Gemini line has led since, reaching the mid-70s with Gemini 3.5 Flash by mid-2026.

The gap that remains is the point of the benchmark. Embodied reasoning asks for something more than image captioning or visual question answering: a model has to build a usable spatial model of a scene and reason about interacting with it, the same competence a robot policy would need. That the strongest general-purpose models still miss roughly a quarter of ERQA’s questions is a reminder that fluent scene description and reliable physical reasoning are not the same capability.