Benchmarks · Agents, tools & computer use
AutomationBench
Can an AI agent carry out a real business workflow across multiple SaaS applications — finding the right API endpoints itself, following a company's own layered business rules, and getting the right data to the right system — rather than completing a single, well-specified task?
ZapierReleased 21 April 2026Live
Business automation is usually pitched as connecting one app to another. AutomationBench, built by Zapier researchers Daniel Shepard and Robin Salimans, tests something closer to what that actually requires in practice: an agent dropped into a simulated company with dozens of SaaS tools, asked to complete a workflow — routing a lead, processing an expense, closing a support ticket — without being told which tool exposes which API. The agent has to find the right endpoints itself, follow the business’s own layered approval rules, and avoid being thrown off by the irrelevant or misleading records that fill a real system.
Drawing on patterns from Zapier’s own automation platform, the benchmark spans six domains — Sales, Marketing, Operations, Support, Finance and HR — with around 1,200 tasks split between a public set and a larger, harder private set reserved for the official leaderboard. Scoring is entirely programmatic: a task counts as solved only if the correct data ends up in the correct place, with no partial credit for a plausible-looking but incorrect action.
The results at release were a sharp contrast to the near-saturated benchmarks common elsewhere in agentic evaluation: every frontier model scored under 10%, with Claude Opus 4.7 leading at 9.9%, Gemini 3.1 Pro close behind at 9.6%, and GPT-5.4 at 7.6%. That gap between models’ fluency on short, well-specified coding and reasoning tasks and their fragility on multi-step, real-world business workflows is precisely what AutomationBench was built to expose.
The set
Around 1,200 tasks (600+ public, 600+ held-back private) spanning six business domains — Sales, Marketing, Operations, Support and HR — built from real workflow patterns observed on Zapier's automation platform across 47 simulated tools. Agents must discover relevant API endpoints themselves, respect business-policy constraints, and work in environments seeded with irrelevant or misleading records; scoring is fully programmatic, checking only whether the correct end state was reached.
Example
There's a scheduling conflict on February 20, 2026 at 2:00 PM — a Zoom meeting and a Google Calendar event overlap. Check the meeting priority policy in the spreadsheet to determine which one wins, then reschedule the loser by prepending [RESCHEDULED] to its topic/title. Post a summary to #ops-updates on Slack noting which meeting won and which was rescheduled, including both the Zoom meeting ID and Calendar event ID.arxiv.org
Where it stands
At release, even frontier models scored under 10% — Claude Opus 4.7 led at 9.9%, ahead of Gemini 3.1 Pro at 9.6% and GPT-5.4 at 7.6% — making it one of the least-saturated agentic benchmarks of its period.