τ-bench

τ-bench, built by the customer-service AI company Sierra and released in mid-2024, tests something most agent benchmarks skip: a conversation with someone who has their own agenda. Its retail and airline domains give an agent real backend tools and a written policy document — the same kind of rules a human support agent would follow — and then run it against a simulated customer who can change the request mid-conversation, push back, or ask something the policy does not clearly cover. Success is measured against the database’s actual end-state, not the words the agent used, and a “pass^k” metric specifically scores how consistently an agent succeeds across repeated attempts at the same task, rather than crediting a single lucky run.

Early results showed the reliability problem plainly: GPT-4o solved close to 61% of retail tasks on a single try but its repeat-trial consistency collapsed, with fewer than a quarter of tasks solved correctly across eight attempts in a row. That reliability gap, more than raw task-completion rate, became the benchmark’s most-cited finding, and by early 2025 frontier labs were citing τ-bench scores as evidence of agentic dependability in model release announcements, including Anthropic’s Claude 3.7 Sonnet.

Sierra has kept extending the benchmark faster than most: a 2025 successor, τ²-bench, added a telecom domain where the simulated user can also act on the shared environment rather than just describing a problem, and a further 2026 revision added banking and knowledge-retrieval tasks plus voice evaluation. By March 2026 leading models were clearing close to 90% on τ²-bench’s text tasks, a sign of how quickly labs converged on structured tool-use and policy-following once it became a benchmark worth optimising for.