ScreenSpot

Before an AI agent can operate a computer, it has to find things on the screen: the right button, the right field, the right icon. ScreenSpot isolates that skill, separate from the harder problem of planning a whole multi-step task. Introduced by researchers at Nanjing University, with Shanghai AI Laboratory and the National University of Singapore, alongside the SeeClick GUI agent in January 2024, it pairs more than 1,200 natural-language instructions with over 600 screenshots across iOS, Android, macOS, Windows and the web. A model passes a question if its predicted click point lands inside the correct element.

At launch, general-purpose vision-language models were poor at this even when capable elsewhere: GPT-4V managed 16.2% and Qwen-VL just 5.2%, while GUI-specialised models did far better — CogAgent reached 47.4% and SeeClick itself, despite fewer parameters, led at 53.4%. The gap showed that GUI grounding was not a byproduct of general visual competence but needed its own training data, which SeeClick built by automatically mining click targets from web and mobile interfaces.

As grounding models improved, the original set stopped discriminating between them, and the field moved to harder variants: a cleaned-up ScreenSpot-v2, and ScreenSpot-Pro, released in April 2025 to test grounding on professional software — CAD tools, IDEs, scientific applications — at native high resolution. At its release, the best existing model solved only 18.9% of ScreenSpot-Pro’s tasks, showing how far grounding on cluttered professional interfaces lagged behind the near-saturated original benchmark. ScreenSpot’s lineage now sits behind most reported grounding numbers for agentic browser- and computer-use systems.