Separating signal from noise in coding evaluations
OpenAI audited SWE-Bench Pro and found roughly 30% of its tasks are broken due to strict, underspecified, low-coverage, or misleading test setups.
It shows a widely-adopted successor benchmark is itself unreliable, so reported coding-capability gains built on it may not be real.
OpenAI — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
“Through a detailed audit, we find widespread task issues in SWE-Bench Pro and estimate that ~30% of the tasks are broken.”
OpenAI ran a two-part audit, an automated pipeline plus a human annotation campaign, on the SWE-Bench Pro coding benchmark and found that a large share of tasks are flawed in ways that penalize correct model solutions or let incomplete ones pass. As a result, OpenAI retracts its earlier recommendation to adopt SWE-Bench Pro.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
Overly strict tests
“enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.”Introduction
In plain terms: Tests that reject correct solutions just because they were implemented differently than the reference answer.
Underspecified prompts
“omit requirements that hidden tests enforce and that are not reasonably inferable.”Introduction
In plain terms: Task instructions that leave out requirements the hidden grading tests actually check for.
Low-coverage tests
“under check the requested feature, so incomplete fixes can pass.”Introduction
In plain terms: Tests too weak to catch an incomplete fix, so a partial solution can still pass.
Misleading prompt
“points models toward the wrong behavior or contradicts what tests require.”Introduction
In plain terms: A task description that actively steers the model toward behavior the grading tests will not accept.
Gold patch
“the ground-truth reference solution (known as the gold patch)”Human annotation campaign
In plain terms: The accepted correct code change that a task's grading is built around.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Built an automated data-quality pipeline to flag likely broken tasks.
Trace this step to the paper
“An initial automated filter reviews the instructions given to the model, attempts by the model to solve the task, and the tests used to grade these attempts to flag likely broken or problematic examples.”Methodology
- Ran the filter over the benchmark, producing an initial pool of suspect tasks.
Trace this step to the paper
“This filter flagged 286 potentially broken tasks.”Methodology
- Audited each flagged task with Codex-based investigator agents given repository access.
Trace this step to the paper
“Each flagged problem is audited with Codex-based investigator agents that were given access to the task repository and environment.”Human-supervised agent review
- Had investigator agents run tests and inspect repo files/model failure modes across several independent repeats, then had a researcher issue a final judgment.
Trace this step to the paper
“The agent can run tests, inspect files in the repo, and investigate model attempts and their common failure modes on the task. After several independent repeats of these deeper audits, a researcher reviewed the summaries, made a final judgment, and labeled the likely issues.”Human-supervised agent review
- In parallel, ran a human annotation campaign with trained software engineers reviewing the same flagged subset.
Trace this step to the paper
“In parallel, we ran a human annotation campaign over the flagged subset. We worked with experienced software engineers who were trained on the benchmark goals, issue taxonomy, and edge cases before reviewing tasks. Each task was reviewed by five engineers.”Human annotation campaign
- Had reviewers form independent judgments from the problem statement, tests, and gold patch before consulting pipeline output, then assign labels and escalate disagreements.
Trace this step to the paper
“Reviewers formed an independent judgment from the visible problem statement, test cases, and the ground-truth reference solution (known as the gold patch) before using the pipeline analysis or transcript as supporting context. The reviewers then assigned a label and severity rating based on concrete evidence, and escalated disagreements or low-confidence cases for further review.”Human annotation campaign
- Compared agent-pipeline labels against human labels to measure agreement and identify where the pipeline undercounted issues.
Trace this step to the paper
“Of the categories the agent pipeline flagged, reviewers’ judgments overlapped in 74% of cases.”Human annotation campaign
Exactly what was run, and how
| Model | Developer | Temp | Effort / reasoning | Deployment | Other settings |
|---|---|---|---|---|---|
| Codex-based investigator agents | OpenAI | not reported | not reported | unstated | — |
Source for Codex-based investigator agents settings
“Each flagged problem is audited with Codex-based investigator agents that were given access to the task repository and environment.”Human-supervised agent review
What they reported — and what they left out
The post names only the Codex-based investigator agents used to audit tasks, with no temperature, reasoning-effort, or deployment settings given, and never identifies which specific frontier models were the ones evaluated on SWE-Bench Pro itself.
The numbers they report
On the benchmark's public task split, model pass rates rose sharply over eight months, motivating scrutiny of what that gain actually reflects.
731-task public split; pass rate 23.3% to 80.3% in eight months
See it in the paper
“On the 731-task public split, frontier models improved from a pass rate of 23.3% to 80.3% in eight months.”Introduction
An initial automated filter flagged a sizeable pool of tasks for deeper review.
286 potentially broken tasks flagged
See it in the paper
“This filter flagged 286 potentially broken tasks.”Methodology
Both the automated pipeline and the human review found a large share of the flagged tasks were genuinely broken, with humans finding more.
200 tasks (27.4%) via pipeline; 249 tasks (34.1%) via human annotation
See it in the paper
“Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%).”Introduction
The two review methods agreed on category labels for most, but not all, flagged tasks.
74% overlap in category judgments
See it in the paper
“Of the categories the agent pipeline flagged, reviewers’ judgments overlapped in 74% of cases.”Human annotation campaign
Low-coverage tests were the issue type with the biggest gap between human and pipeline detection rates.
9.4% (human) vs 4.1% (agent pipeline) for low-coverage tests
See it in the paper
“The largest difference was in low-coverage tests, which humans selected as the most common issue for 9.4% of the benchmark compared with 4.1% from the agent pipeline.”Human annotation campaign
OpenAI's headline estimate of the benchmark's overall broken-task rate.
~30% of SWE-bench Pro tasks estimated broken
See it in the paper
“we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results.”Introduction
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
SWE-Bench Pro contains widespread broken tasks, at an estimated rate of about 30%.
“we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results.”
“Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%).”
IntroductionThe agent-plus-reviewer pipeline is conservative, undercounting issues relative to human judgment.
“This suggests the agent-plus-reviewer pipeline resulted in conservative labeling: it captured the same broad failure modes humans identified, while undercounting cases where reviewers saw additional or overlapping issues.”
“Compared with the agent pipeline, the human reviewers were also more likely to select multiple labels for a task, indicating that they found tasks to be broken in multiple ways or did not fit cleanly into a single category.”
Human annotation campaignEvaluation flaws are now easier to detect than they used to be, because models themselves can be used to inspect benchmarks.
“At the same time, evaluation flaws are easier to detect now than they would have been even a short time ago.”
“As model capabilities improve, we can use those models to inspect prompts, tests, patches, traces, and edge cases with much greater depth and consistency, helping surface benchmark issues that were previously costly or impractical to find at scale.”
DiscussionOpenAI now retracts its own earlier recommendation that the field adopt SWE-Bench Pro.
“Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.”
“We find evidence of breaking issues in a significant portion of the dataset. Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%).”
IntroductionHow they frame it, and what they want next
Their framing
OpenAI frames the finding not as an indictment of SWE-Bench Pro's creators but as a structural problem: benchmarks sourced from real open-source pull requests inherit messiness from human collaboration that makes clean, isolated grading hard. They pair this critique with an optimistic turn, presenting the audit itself as evidence that stronger models make it newly practical to catch such flaws at scale.
Register: The post states its central finding and the retraction of its own prior recommendation plainly and without much hedging, while using softer language like "estimate" and "suggests" around specific percentages and interpretive claims.
Where they hedge
“we estimate that ~30% of SWE-bench Pro tasks are broken”Introduction
“This suggests the agent-plus-reviewer pipeline resulted in conservative labeling”Human annotation campaign
What they say it means
- Flawed benchmarks can distort safety and deployment decisions that rely on them.
the paper’s words
“When evaluations have flaws that affect results, they can give a false understanding of capabilities, misrepresenting safety cases and affecting research priorities.”Introduction
- The field should move toward benchmarks purpose-built and vetted by experienced engineers rather than mined wholesale from open-source history.
the paper’s words
“We hope the wider evaluation community will develop new benchmarks built by experienced software developers specifically to test model capabilities.”Discussion
What they call for next
- Model developers should carefully scrutinize results reported on SWE-Bench Pro rather than take them at face value.
the paper’s words
“we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results.”Introduction
- The evaluation community should build new, developer-vetted benchmarks with better human oversight built into the process.
the paper’s words
“We hope the wider evaluation community will develop new benchmarks built by experienced software developers specifically to test model capabilities. That approach can preserve the high bar and realism we want to measure model capabilities, and allows for better human oversight throughout the process.”Discussion
Limitations they state
“There was also some disagreement on categories between the two review paths, but in no flagged task was “not broken” the most common human label.”Human annotation campaign
Moves worth stealing
Opens by grounding a technical audit in institutional stakes (safety and deployment decisions) before presenting any findings.
“Accurately measuring our models’ capabilities is important for sound deployment and safety decisions, including decisions under OpenAI’s Preparedness Framework”
States a self-critical reversal of the company's own prior public recommendation directly, without softening or burying it.
“Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.”
Reframes a negative finding about benchmark quality as positive evidence of growing model/agent capability.
“As model capabilities improve, we can use those models to inspect prompts, tests, patches, traces, and edge cases with much greater depth and consistency, helping surface benchmark issues that were previously costly or impractical to find at scale.”
Appends a concrete, single-task worked example after the general discussion to make an abstract failure category tangible.
“In several cases the task prompt prescribed a specific implementation, but the hidden test cases expected different behavior.”
Where else this leads
Same people
- Introducing GeneBench-Pro OpenAI
shares OpenAI - Where the goblins came from OpenAI
shares OpenAI
Published alongside it
The nearest publications in time, across all three labs.
- Quantifying the Salience of Geo-Cultural Values for Pluralistic Safety Alignment Google DeepMind
2026-07-10 - The Case for Globally Beneficial Technology Google DeepMind
2026-07-06 - Towards Structural Understanding of LLM Overthinking Google DeepMind
2026-07-02 - Agentic Misalignment in Summer 2026 Anthropic
2026-07-15
What this page was built from
Working text is a scraped copy of OpenAI's blog post (corpus manifest grades it 'partial'), including site navigation boilerplate and one appended worked example (the OpenLibrary task) rather than a clean article-only capture; this is a short blog post, not a full paper, so several schema sections (e.g. constructs, models_studied) are thin by nature of the source rather than by omission.