Separating signal from noise in coding evaluations

Accurately measuring the capabilities of AI models is crucial for sound deployment and safety decisions. When evaluation benchmarks have flaws, they can provide a false understanding of model capabilities, affecting safety cases and research priorities. OpenAI previously investigated SWE-bench Verified and found fundamental design flaws, leading them to recommend SWE-Bench Pro.

SWE-Bench Pro was designed to improve coding task evaluations by testing models on longer horizons and more realistic coding tasks. Despite initial improvements in model performance on this benchmark, a recent audit revealed widespread issues. OpenAI’s audit, involving a datapoint analysis pipeline and review by experienced software engineers, found that approximately 30% of the 731 tasks in SWE-Bench Pro are broken.

These broken tasks primarily fell into four categories, indicating problems with test cases expecting different behavior than prescribed, or low-coverage tests. This emphasizes the difficulty of curating hard but fair benchmarks for AI models. The findings underline the need for rigorous quality assurance in evaluation datasets.

To address this, OpenAI developed a quality assurance pipeline. This pipeline uses an initial automated filter to flag problematic examples, followed by a deeper agent-assisted audit and a human annotation campaign involving experienced software developers. This multi-layered approach helps distinguish between reasonable task ambiguity and true underspecification, ensuring task failures genuinely reflect model limitations and successes represent valid solutions.