SWE-Lancer

Most coding benchmarks report a pass rate. SWE-Lancer, which OpenAI released in February 2025, reports a dollar figure instead: it drew more than 1,400 real freelance software-engineering jobs from Upwork, worth roughly $1 million in payouts clients had actually agreed to, and asked how much of that money a model could actually earn. Coding tasks were graded with end-to-end tests verified by experienced engineers, and a separate “managerial” track tested something different — whether a model could pick the better of two competing technical proposals, scored against the choice the human hiring manager on the job had really made.

The launch results were unflattering to the field generally rather than to any one lab. Of the three models OpenAI tested — its own GPT-4o and o1, and Anthropic’s Claude 3.5 Sonnet — Sonnet performed best, earning about $400,000 of the $1 million on offer and resolving roughly a quarter of individual coding tasks. OpenAI’s own summary was direct: “the majority of its solutions are incorrect, and higher reliability is needed for trustworthy deployment.” Framing capability in dollars a client had actually paid, rather than an abstract percentage, made the gap between benchmark performance and professional-grade engineering unusually concrete, and the result became a frequently cited counterpoint whenever other benchmarks suggested AI coding tools were closing in on human engineers.

OpenAI reported against the benchmark through 2025, though generally using a task-pass-rate metric for the coding split rather than the original dollar figure — GPT-5.1-Codex-Max reported resolving 79.9% of independent-contractor tasks in November 2025, up from 66.3% for its immediate predecessor a week earlier, illustrating how quickly frontier coding agents closed a gap that had looked wide at launch. That November figure is, however, the last word on it: no other lab adopted SWE-Lancer — Anthropic and Google report SWE-bench Verified instead — and OpenAI itself dropped it from its subsequent system cards, replacing it in the GPT-5.5 card with a new “Monorepo-Bench”. So SWE-Lancer captured a real gap vividly at launch but never became the shared, live leaderboard that SWE-bench did.