ChartQA
Can a model answer questions about a bar, line or pie chart that require reading values off it and then performing a logical or arithmetic operation on them?
York University with NTU Singapore & Salesforce Research (Masry, Hoque, Joty et al.)Released 19 March 2022
Reading a chart and answering a question about it sounds simple, but it packs together two separate skills: seeing the chart accurately, and doing the right arithmetic once you have. ChartQA, built by Ahmed Masry and Enamul Hoque at York University with collaborators from NTU Singapore and Salesforce Research, was designed to test both at once. Its questions are drawn from real bar, line and pie charts published by outlets such as Pew Research and Statista, split between 9.6K written by humans and a larger machine-generated set built from human-authored chart summaries, so that answering correctly often means comparing two bars or summing a series rather than simply looking up a labelled value.
The benchmark was folded early into how major labs reported vision capability: both OpenAI’s GPT-4 announcements and Google’s Gemini technical reports cited ChartQA scores as a standard measure of chart understanding. Progress was fast — by June 2024, Claude 3.5 Sonnet reported 90.8% relaxed accuracy on the test set, ahead of GPT-4o’s 85.7% and Gemini 1.5 Pro’s 87.2% on the same comparison table, and Alibaba’s Qwen2.5-VL-72B reported a close 89.5% in its own February 2025 technical report.
Scores clustering above 85–90% across several labs by mid-2024 pointed to saturation, and in 2025 a separate research team introduced ChartQAPro, explicitly built to be harder — adding infographics, dashboards and a wider range of question formats — after finding that modern vision-language models had largely closed the gap on the original set. ChartQA itself remains widely cited as a standard chart-reading component in multimodal model releases even as attention shifts toward its successor.
The set
9.6K human-written questions plus 23.1K questions machine-generated from human-authored chart summaries, over real-world charts drawn from sources such as Pew Research and Statista. Many questions require more than lookup — comparing two values, summing a series, or computing a difference — and are graded with 'relaxed accuracy', which tolerates small numeric rounding rather than demanding an exact string match.
Example
Which year has the most divergent opinions about Brazil's economy? (Answer: 2015)arxiv.org
Where it stands
Reported scores were already above 90% for some frontier models by mid-2024 (Claude 3.5 Sonnet, 90.8%), and a harder successor, ChartQAPro, was introduced in 2025 specifically because ChartQA was judged to be saturating against modern vision-language models.
How the top score changed hands
- June 2024Claude 3.5 Sonnet / GPT-4o / GPT-4 Turbo / Gemini 1.5 Pro90.8% / 85.7% / 78.1% / 87.2% (test, relaxed accuracy)From Anthropic's own model card addendum comparison table.
- February 2025Qwen2.5-VL-72B89.5% (test avg)Reported in Qwen2.5-VL's own technical report, alongside GPT-4o at 86.7% and Claude 3.5 Sonnet at 90.8% on the same table.
Current best: Claude 3.5 Sonnet — 90.8% (test, relaxed accuracy) Most recent score independently confirmed from an official model card, ahead of GPT-4o (85.7%) and Gemini 1.5 Pro (87.2%) on the same table; Qwen2.5-VL-72B reported a lower 89.5% eight months later, so this remains the highest reliably-sourced figure found for this entry.