Andy Konwinski launches $1M Konwinski Prize for contamination-free SWE benchmark

Andy Konwinski, a co-founder of Databricks and Perplexity, pledged $1 million to the first open-source AI system to score above 90% on a new, contamination-resistant version of SWE-bench, the benchmark that scores models on real GitHub issues.

Konwinski’s objection to the existing benchmark was structural rather than about any one model’s score: SWE-bench draws its problems from public repositories, and because those same repositories are widely used as training data, a model’s reported performance could reflect memorisation rather than problem-solving. His prize, run on Kaggle and restricted to open-source, open-weight entrants, addressed this by collecting its test set of GitHub issues only after the submission deadline, so no model could have been trained on the specific problems it would be scored against. A baseline reward of at least $50,000 was guaranteed to the top submission even if no one reached the 90% threshold.

The first competition closed in mid-2025, and its result underlined the concern that motivated it: the winning entry scored 7.5% accuracy, against roughly 75% that models routinely posted on the original SWE-bench Verified subset and 34% on the harder full set. Konwinski said the gap left him unsure whether it reflected contamination in the older benchmark or simply how much harder unseen, real-world engineering problems were than the ones models had been evaluated against before, but called the result “a reality check.”