OpenAI releases SWE-Lancer benchmark

OpenAI released SWE-Lancer, a benchmark built from more than 1,400 real freelance software-engineering tasks drawn from Upwork, together worth about $1 million in the payouts clients had actually agreed to pay for them — from $50 bug fixes to a $32,000 feature build. Independent coding tasks were graded with end-to-end tests triple-verified by experienced engineers; a separate “managerial” category tested whether a model could pick the better of two competing technical proposals, scored against the choice the human hiring manager on the job had actually made.

The accompanying paper evaluated OpenAI’s GPT-4o and o1 alongside Anthropic’s Claude 3.5 Sonnet. Claude 3.5 Sonnet was the strongest of the three, earning roughly $400,000 of the $1 million on offer and correctly resolving about a quarter of individual coding tasks, ahead of o1 and GPT-4o. It did comparatively better on the managerial-selection tasks than on hands-on implementation. OpenAI’s own assessment was blunt: even for the leading model, “the majority of its solutions are incorrect, and higher reliability is needed for trustworthy deployment.”

By pricing tasks in the dollars a client had actually agreed to pay, rather than an abstract pass rate, SWE-Lancer let capability be read directly against a professional benchmark, and the resulting gap — models solving a minority of paid, real-world engineering work — became a frequently cited counterpoint to benchmark scores suggesting AI coding tools were close to matching professional engineers. OpenAI also released a public “Diamond” subset and a Docker evaluation environment so other labs could test their own models against the same tasks.