Summary
OpenAI’s Preparedness team, with the original SWE-bench authors, found that SWE-bench systematically underestimates coding agents because some tasks are hard or impossible to solve for reasons unrelated to the model. They name three causes: unit tests that are overly specific or unrelated to the issue, issue descriptions that are underspecified, and development environments that are hard to set up reliably, so valid patches fail anyway. 93 Python developers annotated 1,699 random test samples, each three times, for problem-statement specification, FAIL_TO_PASS test validity and difficulty; the most severe of the three labels was kept. The result is SWE-bench Verified, 500 samples that supersede the original test set and SWE-bench Lite, released with a new containerized Docker harness and the full annotations. Source is a blog post (August 2024, updated February 2025), not a paper.
Key points
- Three ways SWE-bench mis-scores a correct solution: overly specific unit tests (one example requires an exact deprecation message the issue never mentions), underspecified issue descriptions, and unreliable environment setup that fails tests regardless of the patch.
- Annotation campaign: 93 developers experienced in Python, gated by onboarding tests built from 50 samples the team hand-labelled; 1,699 samples, each labelled by 3 separate annotators on a 0–3 severity scale, with the highest severity taken (“conservatively ensemble”). Labels 2 and 3 discard the sample.
- 38.3% of samples were flagged for underspecified problem statements and 61.1% for unit tests that may unfairly reject valid solutions; 68.3% were filtered out overall. The authors call the filter likely “overzealous” but high-confidence.
- 500 verified samples, with difficulty annotations: an “easy” subset of 196 tasks under 15 minutes and a “hard” subset of 45 tasks over an hour.
- GPT-4o’s best scaffold scores 33.2% on Verified against 16% on the original SWE-bench. Performance rises within each difficulty bucket, which the authors read as removing impossible samples rather than just shifting towards easier ones.
- Limitations they name: static datasets scraped from public GitHub repos are likely to be contaminated for models pre-trained on internet text, and SWE-bench covers only a narrow slice of autonomous capability.
Relevance
The canonical example of repairing a benchmark’s reward signal by human verification, and the template the later “Verified” efforts follow (OSWorld-Verified). For Benchmark Validity it quantifies how much of a popular benchmark’s failure rate was the benchmark’s fault. For WorldSmith it is the human-cost baseline any automated verifier audit has to beat: three developer passes per sample to keep 500 of 1,699. BenchGuard is the attempt to do the same audit with LLMs.
Notes
From paper club, 19 May 2026:
While SWE-Bench Verified is deprecated and saturated/contaminated today (models have seen the OSS code and can now recall exact fixes), it has some utility in its core ideas. Even <2 years later, the well-specified short tasks that relied on human annotators can be handled by LLMs-as-judges, which is how we will approach this type of problem.