VQAv2

The original VQA dataset, released in 2015, had a well-known flaw: models could score well by exploiting patterns in a question’s wording alone, without looking at the image — a question starting “Is there a…” was usually answered “yes” regardless of what the picture showed. Researchers at Virginia Tech, Georgia Tech and the US Army Research Laboratory built VQA v2.0 to close that loophole. Every question was paired with two visually similar images chosen to have different correct answers, so a model relying on language priors rather than genuine image understanding would be penalised. The result was roughly 1.1 million question-image pairs over 204,000 COCO photographs, with about 11 million answers collected from crowd workers.

The balancing worked as intended: the paper reported that state-of-the-art models of the time performed markedly worse on the rebalanced questions than on the original set, confirming how much of their apparent performance had been linguistic pattern-matching. The best baseline tested, a bilinear-pooling model, reached 62.27% overall accuracy against a 44.26% language-only baseline.

VQAv2 became the standard general-purpose visual question answering benchmark through the 2010s and into the era of large vision-language models, cited in evaluation suites from early attention-based architectures through GPT-4V-class multimodal LLMs. By the early 2020s, leading models were approaching human-level accuracy on it, and it has since faded from frontier model reports in favour of benchmarks built to resist saturation. In 2025 the VQA challenge series’ creators received the Mark Everingham Prize, recognising the series for opening up vision-and-language research as a field.