Introducing GeneBench-Pro
OpenAI's GeneBench-Pro benchmark tests whether AI agents can handle ambiguous, judgment-heavy computational biology research, and its best model still solves fewer than a third of the problems.
It shows concretely how far frontier models still are from independently handling the ambiguous, judgment-heavy work of real scientific research.
OpenAI — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
“A research-level benchmark measuring how AI agents navigate ambiguity and make consequential judgments in computational biology.”
OpenAI built a harder, more realistic benchmark of 129 computational-biology problems to test whether AI agents can handle the ambiguous, judgment-heavy parts of research, not just execute a known workflow. Its strongest model currently solves under a third of the problems even with extra reasoning effort, though that is a large jump from where the original version of the benchmark started.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
Research taste
“we define “research taste” as the chains of judgment calls that shape an analysis: which questions the data can support, how early diagnostics should change the model or estimand, and when an initial plan needs to be revised.”Introducing GeneBench-Pro (opening overview)
In plain terms: The chain of judgment calls a researcher makes about what the data can support, when to change approach, and when a plan needs revising.
Net clinical utility (case-study estimand)
“Report net clinical utility = benefit risk difference (percentage points) - 0.35 * toxicity risk (percentage points), and choose therapy_class_code 1 if TXR1i has positive net utility and 0 otherwise.”Case study: Structural variant-guided tumor therapy benefit-risk decision
In plain terms: A single score for a sample problem that subtracts a weighted toxicity risk from a treatment's clinical benefit to decide whether the therapy is worth recommending.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Build each problem synthetically with a fully known causal structure, directly simulating the data-generating process instead of using unresolvable historical data.
Trace this step to the paper
“To avoid these failure modes, each GeneBench-Pro problem is built synthetically: we know the full causal structure and directly simulate the data-generating process.”Dataset construction
- Tune each problem's complexity and run ablation studies confirming that reasonable analytical variation still passes while plausible-but-incorrect analyses fail.
Trace this step to the paper
“That enables us to tune the complexity of each problem, ensure that reasonable differences in subjective analytical choices still produce accepted numerical results, and verify (through ablation studies) that plausible but incorrect analyses fail.”Dataset construction
- Audit problem drafts through detailed trace analyses to check for information leakage and unintended solution shortcuts.
Trace this step to the paper
“We then audit problem drafts through detailed trace analyses to check for information leakage and unintended solution pathways.”Dataset construction
- Send a subset of problems to external domain experts to review realism, answer identifiability, and method appropriateness.
Trace this step to the paper
“We sent 82 of the 129 GeneBench-Pro questions to external domain experts, including graduate students, postdoctoral researchers, industry scientists, and professors. Reviewers assessed each problem’s realism, whether the target answer was identifiable, and whether the methods and estimators were appropriate.”Dataset construction
- Give each agent an isolated workspace with a short prompt, data files, and a standard bioinformatics software stack to work in.
Trace this step to the paper
“Agents receive access to an isolated workspace with a short prompt, data files, and a standard bioinformatics stack including Python, scientific computing libraries, and basic genomics packages like PLINK 2.0”Evaluation and grading
- Grade agent answers deterministically against known targets, since the full data-generating process is controlled, rather than using rubric-based scoring.
Trace this step to the paper
“Because we control the full data-generation process, we can grade correctness deterministically against known targets, avoiding model-choice variability and verbosity effects found in standard rubric-based evaluation.”Evaluation and grading
- Evaluate the same model at multiple reasoning-effort levels (lowest vs. highest) to measure the effect of scaling test-time compute.
Trace this step to the paper
“At the lowest reasoning level, GPT‑5.6 Sol only achieves a single-digit passrate.”Results
- Open-source a representative subset of questions with an interactive interface and route a further subset to a third party for independent benchmarking.
Trace this step to the paper
“We are fully open-sourcing 10 representative GeneBench-Pro questions on Hugging Face (opens in a new window) , with an interactive web interface for browsing them.”Evaluation and grading
Exactly what was run, and how
| Model | Developer | Temp | Effort / reasoning | Deployment | Other settings |
|---|---|---|---|---|---|
| GPT-5.6 Sol | OpenAI | not reported | Tested at multiple reasoning levels: lowest (single-digit pass rate) and highest (28.7% pass rate), plus an optional Pro mode (31.5%) | unstated | — |
| GPT-5.2 | OpenAI | not reported | not reported | unstated | — |
| GPT-5 | OpenAI | not reported | not reported | unstated | — |
| GPT-5.5 | OpenAI | not reported | not reported | unstated | — |
| GLM 5.2 | — | not reported | not reported | unstated | — |
Source for GPT-5.6 Sol settings
“Our strongest model at the time of testing, GPT‑5.6 Sol, attains a pass rate of 28.7% at the highest reasoning level (31.5% with Pro mode enabled).”Results
Source for GPT-5.2 settings
“solves nearly six times as many questions as GPT‑5.2 does while using about two-thirds as many tokens.”Results
Source for GPT-5 settings
“at that time, our best frontier model, GPT‑5, scored below 5%.”Results
Source for GPT-5.5 settings
“The performance gap between GPT‑5.6, GPT‑5.5 and leading open-source models such as GLM 5.2 is significantly larger than we would expect when extrapolating from coding benchmarks”Results
Source for GLM 5.2 settings
“and leading open-source models such as GLM 5.2 is significantly larger than we would expect when extrapolating”Results
What they reported — and what they left out
The post names the specific models compared and reports pass rates and relative token usage at different reasoning levels, but gives no temperature, sampling, prompt, or deployment-mode details for any model, and never states who built the comparison model GLM 5.2.
The numbers they report
OpenAI's best model, GPT-5.6 Sol, passes under a third of GeneBench-Pro problems even at its highest reasoning setting.
28.7% pass rate at highest reasoning level; 31.5% with Pro mode enabled
See it in the paper
“Our strongest model at the time of testing, GPT‑5.6 Sol, attains a pass rate of 28.7% at the highest reasoning level (31.5% with Pro mode enabled).”Results
Progress since the original GeneBench has been dramatic, going from under 5% to nearly a third of problems solved.
GPT-5 scored below 5% on the original GeneBench; GPT-5.6 Sol scores 28.7% (31.5% with Pro mode) on GeneBench-Pro
See it in the paper
“Our strongest model at the time of testing, GPT‑5.6 Sol, attains a pass rate of 28.7% at the highest reasoning level (31.5% with Pro mode enabled). That is a sharp increase from when we began building the original GeneBench; at that time, our best frontier model, GPT‑5, scored below 5%.”Results
Scaling test-time reasoning effort has a large effect: the same model jumps from single digits to 28.7% pass rate depending on reasoning level.
Single-digit pass rate at lowest reasoning level vs. 28.7% at highest reasoning level
See it in the paper
“At the lowest reasoning level, GPT‑5.6 Sol only achieves a single-digit passrate.”Results
At the highest reasoning level, the newer model solves roughly six times as many questions as its predecessor while using fewer tokens.
~6x as many questions solved as GPT-5.2, using about two-thirds as many tokens
See it in the paper
“solves nearly six times as many questions as GPT‑5.2 does while using about two-thirds as many tokens.”Results
Leading open-source models lag the top GPT models on this benchmark by more than their coding-benchmark scores would predict.
See it in the paper
“The performance gap between GPT‑5.6, GPT‑5.5 and leading open-source models such as GLM 5.2 is significantly larger than we would expect when extrapolating from coding benchmarks”Results
Despite GPT models being used to build and harden the benchmark, competitor models did not outperform the era-matched GPT model and mostly fell behind it.
See it in the paper
“However, competitor models at best matched the performance of the corresponding GPT model at the time of release, and tended to fall short considerably.”Results
A single GeneBench-Pro problem is estimated to cost thousands of dollars in expert time to solve by hand, versus only a few dollars of AI inference cost to attempt.
20-40 hours of human expert time per problem at a conservative $200/hour (thousands of dollars), vs. inference cost of only several dollars per problem
See it in the paper
“our reviewers estimated that a typical GeneBench-Pro problem would take a human expert around 20–40 hours to complete. At a conservative $200 per hour, that puts the human labor cost of a single problem in the thousands of dollars. Current AI agents are still too unreliable to replace human experts, but the cost gap is large, with inference costs at only several dollars per problem.”Results
GeneBench-Pro spans a broad range of computational biology, organized into 10 domains and 21 sub-domains of problems.
129 problems; 10 domains; 21 sub-domains; 82 of 129 sent to external reviewers
See it in the paper
“Domain Atlas: 129 problems in 10 domains and 21 sub-domains”Dataset construction, Domain Atlas
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
GeneBench-Pro's synthetic construction avoids the arbitrary-cutoff and numerically-insensitive failure modes common to other long-horizon biology benchmarks.
“To avoid these failure modes, each GeneBench-Pro problem is built synthetically: we know the full causal structure and directly simulate the data-generating process. That enables us to tune the complexity of each problem, ensure that reasonable differences in subjective analytical choices still produce accepted numerical results, and verify (through ablation studies) that plausible but incorrect analyses fail. We then audit problem drafts through detailed trace analyses to check for information leakage and unintended solution pathways. This gives us confidence that getting the right answer depends on choosing the correct analytic pathway and not on exploiting a shortcut or matching an arbitrary author preference.”
“verify (through ablation studies) that plausible but incorrect analyses fail.”
Dataset constructionOpenAI worried the benchmark could be biased in favor of GPT models (since GPT models were used to build it), but found competitor models were not disadvantaged.
“we suspected GeneBench-Pro might be biased against GPT models relative to other model families.”
“However, competitor models at best matched the performance of the corresponding GPT model at the time of release, and tended to fall short considerably.”
ResultsGiven the pace of progress, GeneBench-Pro itself may be saturated (solved) by the end of the year.
“At the current pace, this benchmark may be saturated by the end of the year.”
“Our strongest model at the time of testing, GPT‑5.6 Sol, attains a pass rate of 28.7% at the highest reasoning level (31.5% with Pro mode enabled). That is a sharp increase from when we began building the original GeneBench; at that time, our best frontier model, GPT‑5, scored below 5%.”
ResultsExternal reviewer testimony is used to support that GeneBench-Pro problems are genuinely difficult and realistic, not simplified textbook exercises.
“The problems I reviewed would have been challenging for a graduate student to complete without iterated feedback from an experienced supervisor.”
“We sent 82 of the 129 GeneBench-Pro questions to external domain experts, including graduate students, postdoctoral researchers, industry scientists, and professors. Reviewers assessed each problem’s realism, whether the target answer was identifiable, and whether the methods and estimators were appropriate.”
Dataset constructionEven partial automation of this kind of analysis at current AI capabilities could create meaningful economic and scientific value.
“even partial automation at current capabilities could create meaningful economic and scientific value.”
“Current AI agents are still too unreliable to replace human experts, but the cost gap is large, with inference costs at only several dollars per problem.”
ResultsThe size of the performance gap versus leading open-source models indicates those models are more specialized for coding than for broader scientific reasoning.
“indicating that open-source models are more specialized for coding than for broader reasoning ability.”
“The performance gap between GPT‑5.6, GPT‑5.5 and leading open-source models such as GLM 5.2 is significantly larger than we would expect when extrapolating from coding benchmarks”
ResultsHow they frame it, and what they want next
Their framing
OpenAI frames GeneBench-Pro as evidence that frontier models are improving quickly at judgment-heavy scientific reasoning, while still falling well short of independently replacing a human researcher. The post pairs its own upbeat progress numbers and a predicted near-term saturation date with more cautionary notes, several of them voiced through quotes from the outside reviewers who helped build and test the benchmark rather than stated as OpenAI's own headline claim.
Register: The post's own voice is upbeat and forward-leaning about the pace of progress (predicting the benchmark 'may be saturated by the end of the year'), while the sharper caveats about unreliability and shallow data-handling are either brief asides or delivered through quoted outside reviewers rather than the headline framing.
Where they hedge
“Current AI agents are still too unreliable to replace human experts,”Results
“Models can make partial progress on challenging problems, but they struggle to close the inferential loop.”Discussion, after Results
“They aren’t cautious enough about data issues. Maybe that highlights a weakness of current models.”Reviewer testimonial (Lex Flagel, Gencove)
What they say it means
- If agents can reliably automate this class of judgment-heavy analysis, it could meaningfully speed up scientific discovery, since genetic evidence already plays a central role in choosing drug targets.
the paper’s words
“If agents can reliably automate this class of analysis, they could significantly accelerate scientific discovery.”Discussion, closing section
- As sequencing gets cheaper and biobank-scale data grows, models that can reliably do the analysis work of human expert teams could reshape how industrial research iterates on targets and hypotheses.
the paper’s words
“Models that can consistently perform analyses now handled by teams of human experts could transform industrial research by accelerating hypothesis triage, target follow-up, and the iteration cycle between data generation and decision-making.”Discussion, closing section
- As models improve, benchmarks that test higher-level judgment rather than book knowledge or routine execution will matter more.
the paper’s words
“benchmarks that probe model abilities at these higher levels of abstraction will become increasingly useful, beyond those that simply test book knowledge or the ability to execute routine analyses.”Discussion, closing section
What they call for next
- Provide independent, third-party benchmarking of a 50-question GeneBench-Pro subset via an outside evaluator.
the paper’s words
“we will provide a 50-question subset to Artificial Analysis (opens in a new window) for independent, third-party benchmarking in the near future.”Evaluation and grading
Limitations they state
“Current AI agents are still too unreliable to replace human experts,”Results
“the fact that frontier models still solve fewer than a third of these problems shows that there is substantial room for improvement.”Discussion, after Results
“Models can make partial progress on challenging problems, but they struggle to close the inferential loop.”Discussion, after Results
“It seemed like most of the agents failed on (2). They aren’t cautious enough about data issues.”Reviewer testimonial (Lex Flagel, Gencove)
Moves worth stealing
Leads with named, titled external expert testimonials to lend independent credibility to a benchmark the company both built and grades itself.
“The problems I reviewed would have been challenging for a graduate student to complete without iterated feedback from an experienced supervisor.”
States its own potential bias risk explicitly before rebutting it with a comparison, rather than leaving the objection for a reader to raise.
“we suspected GeneBench-Pro might be biased against GPT models relative to other model families.”
Translates a capability result into a dollar figure a business reader can act on, rather than reporting only an accuracy percentage.
“At a conservative $200 per hour, that puts the human labor cost of a single problem in the thousands of dollars.”
Predicts its own benchmark's near-term expiration date given the pace of progress, treating rapid saturation as a marker of success rather than something to hedge around.
“At the current pace, this benchmark may be saturated by the end of the year.”
Illustrates an abstract capability gap with a labeled, side-by-side worked example naming the exact statistical method each named model version used on the same problem.
“Fit a counting-process Cox model with treatment as a time-varying exposure, effective only after treat_start +90 days ... The model included G, treatment×G, baseline severity, age, and sex.”
Where else this leads
Same people
- Separating signal from noise in coding evaluations OpenAI
shares OpenAI - Where the goblins came from OpenAI
shares OpenAI
Same territory
- Discovering cryptographic weaknesses with Claude Anthropic
ai-agents - Introducing LifeSciBench OpenAI
benchmark - SLEIGHT-Bench: Finding Blind Spots in AI Monitors Anthropic
benchmark
Published alongside it
The nearest publications in time, across all three labs.
- Towards Structural Understanding of LLM Overthinking Google DeepMind
2026-07-02 - Bridging the Scale Gap: Augmenting Human Red-Teaming to Uncover Latent Risks in T2I Models Google DeepMind
2026-06-26 - Real-Time Group Dynamics with LLM Facilitation: Evidence from a Charity Allocation Task Google DeepMind
2026-06-26 - Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South Google DeepMind
2026-06-25
What this page was built from
Working from a 'partial'-grade plain-text scrape of the OpenAI blog post, including site navigation and footer boilerplate; the linked full paper (referenced only as 'Read the paper' with no URL text captured), the interactive Domain Atlas widget, and the full case-study sub-pages were not available as text, so any detail that lives only there is not captured here.