Scientific computing in the age of agentic AI
Two titles, both real. The heading above is how the lab announced this work. The document actually behind it is titled “Scientific computing in the age of agentic AI: an exploratory field report” — everything below is read from that document.
Eight field case studies show coding agents (Claude Code, Codex, GPT models) speeding up real scientific-software rewrites, while human effort shifts toward validation rather than disappearing.
It is a rare, candid, multi-team field report on where AI coding agents actually help and fail in real scientific codebases, rather than a benchmark leaderboard score.
Jeremy Li · Alex Rubinsteyn · Sergey Feldman · Timothy O’Donnell · James M. Ferguson · Rob Patro · Ian Driver · Philip A. Ewels · Felix Krueger · Philipp Angerer · Ilan Gold · Jonathan Manning · Lukas Heumos · Mamad Ahangari · … — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
“Scientific computing has become a central component of modern scientific discovery. Yet many computational tools are developed by small, specialized teams under incentives that encourage the release of rapidly prototyped tooling without commensurate attention to engineering concerns, including performance and maintainability. These gaps are particularly visible in the life sciences, where the advent of highthroughput sequencing and molecular profiling has made the production and processing of datasets routine at scales that strain reliability and cost. Recently, LLM-based agents have become increasingly capable, with publicly available systems possessing both significant domain knowledge in many scientific fields and the ability to autonomously operate over complex and specialized codebases in pursuit of well-defined goals. Together, these developments create a practical opportunity for scientific computing. Many of the persistent weaknesses of the scientific computing ecosystem stem from technical debt and a shortage of sustained engineering labor and expertise. Here, we examine coding agents as a potential way to address these weaknesses: we present an exploratory field report of eight early case studies in the application of LLM agents to scientific computing across a range of project scopes, from lightweight maintenance tasks to full performance-oriented rewrites of scientific libraries, with a focus on the life sciences. Each of these case studies is accompanied by reflections from the individual or group responsible for the work, including lessons from the process. Overall, we find that the use of coding agents in scientific computing holds great promise for accelerating scientific research and increasing the reliability of critical systems, but that outstanding concerns remain, including responsibility and ownership for such projects, and we suggest collaboration and stewardship with existing maintainers when feasible.”
The authors report on eight real-world projects where independent teams used LLM coding agents such as Claude Code and Codex to maintain, optimize, port, or rewrite scientific software, mostly in computational biology. They find agents can dramatically cut engineering effort and produce large speedups, but human effort shifts toward defining validation criteria and catching subtle correctness bugs rather than disappearing.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
Byte-level output equivalence vs. logical equivalence
“with certain groups taking the strategy of enforcing byte-level output equivalence, while others sought only to ensure logical equivalence.”Introduction
In plain terms: Two different bars for calling a rewrite 'correct': matching the original tool's output bit-for-bit, versus just agreeing on the values or decisions that matter.
Project scope (surface area affected)
“the raw scope of the changes necessary, meaning the surface area affected”Case Studies
In plain terms: How much of the existing codebase a project touched, from a small patch to a wholesale rewrite.
Validation target
“The validation target depended on the project’s intended relationship to existing behavior.”Case Studies
In plain terms: What a project's correctness was checked against: an existing reference implementation, simulated data, or a fixed acceptance criterion.
Read-ordering metric
“we used a read-ordering metric as a proxy for assembly quality.”Case Study E, Validation and evidence
In plain terms: A stand-in measure of genome-assembly correctness based on whether reads end up in the same order as a trusted baseline assembly, used because exact byte-matching was too strict a bar for a runtime-optimization exercise.
Population-mean agreement criterion
“any scaled gap above 0.25 was considered a failure. This population-mean agreement criterion was used throughout the case study.”Case Study G, Validation and evidence
In plain terms: A pass/fail rule for a statistical software rewrite: each parameter's estimated average must land within one quarter of a reference standard deviation of the original implementation's estimate.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Solicited structured, written case-study submissions from eight independent groups who had already completed agentic coding projects in scientific computing.
Trace this step to the paper
“We present case studies submitted by eight groups who completed agentic coding projects in scientific computing”Case Studies
- Standardized the format of the case studies and checked internal consistency, but explicitly did not independently re-run or validate the contributors' own benchmarks.
Trace this step to the paper
“We standardized their organization and checked internal consistency and selected public artifacts where feasible, but did not independently reproduce every benchmark or validate every reported result.”Case Studies
- Classified the eight projects along three organizing properties: the type of project, the scope of the surface area changed, and what the result was validated against.
Trace this step to the paper
“three properties were useful for organizing the projects: (1) the type of project; (2) the raw scope of the changes necessary, meaning the surface area affected”Case Studies
- Synthesized three recurring cross-case themes: escalating validation burden with project scope, staged/iterative rather than one-shot development, and unreliable agent self-assessment of completion.
Trace this step to the paper
“We identified three recurring themes across all but one case study, reflecting the human’s primary role as a verifier and orchestrator.”Recurring Themes
- In the rustar-aligner case study, validated the Rust reimplementation of STAR by comparing multiple alignment fields against the original tool on a fixed read set.
Trace this step to the paper
“Parity was measured on position, CIGAR, MAPQ, NH tag and proper-pair flag against STAR 2.7.11b on an identical index and arguments, using 10k yeast RNA-seq reads.”Case Study B, Validation and evidence
- In the RustQC case study, validated the rewritten QC pipeline by running the full original nf-core/rnaseq workflow and the rewrite side by side and snapshot-comparing outputs.
Trace this step to the paper
“Eventually I used the nf-core/rnaseq pipeline itself: run it with the original tools, run it again with RustQC, and compare.”Case Study C, Validation and evidence
- In the hifiasm case study, prompted an LLM with profiling results to propose optimizations, screened candidates on a small development dataset, then evaluated survivors on a held-out synthetic benchmark and real human sequencing data.
Trace this step to the paper
“Candidate implementations were tested on a small 12 Mb development dataset to catch compilation failures, performance regressions, and quality problems early.”Case Study E, Agentic intervention
- In the HelixForge case study, isolated the effect of the GPU-native rewrite with matched A/B comparisons using the same donor genome, region, mutation set, and random seed for both tools.
Trace this step to the paper
“Both tools were run on the same donor genome, HG005 from Genome in a Bottle (GIAB) consortium.”Case Study D, Validation of the workflow
- In the bayesm case study, checked the base rewrite against a population-mean agreement criterion, then subjected the two extensions to further convergence diagnostics and simulation-based calibration once initial defects were found.
Trace this step to the paper
“we validated every corrected sampler in this study using the SBC procedure of Talts et al.”Case Study G, Corrected validation and evidence
- Built stylized, explicitly illustrative economic scenarios estimating researcher-time and compute savings from agent-assisted software modernization, using published workload parameters plus stated assumptions.
Trace this step to the paper
“we sketch a few stylized scenarios where we combine published workload parameters with explicit assumptions about adoption, runtime reduction, and unit cost to arrive at rough cost savings estimates.”Economic value of scientific software modernization
Exactly what was run, and how
| Model | Developer | Temp | Effort / reasoning | Deployment | Other settings |
|---|---|---|---|---|---|
| Claude Code and Codex (paired contributor/reviewer) | Anthropic (Claude Code); OpenAI (Codex) | not reported | not reported | agent harness (CLI-based coding agent) | Used in an adversarial pairing pattern: develop with one harness, review with the other, and switch roles when work stalled. |
| Claude Code (Sonnet 4.5/4.6) | Anthropic | not reported | not reported | agent harness (CLI-based coding agent) | — |
| GPT-5.5 Pro (API) and Codex | OpenAI | not reported | not reported | API | — |
| GPT-5.5 | OpenAI | not reported | not reported | unstated | Prompted with profiling results and instructions to optimize runtime while keeping assembly-quality metrics unaffected. |
| GPT-5.2 | OpenAI | not reported | not reported | unstated | Given a brief initial prompt on parity and optimization goals, then largely continued via repeated 'continue' instructions rather than detailed guidance; the authors speculate a newer model would need less intervention. |
| GPT-5.2 then GPT-5.6 (successive optimization passes) | OpenAI | not reported | not reported | unstated | Prompted zero-shot to find optimizations with no constraints on the nature of the changes; essentially no human intervention beyond the initial prompt. |
| Claude 3.5 Sonnet and OpenAI o1 (via aider, early abandoned attempt) | Anthropic (Claude 3.5 Sonnet); OpenAI (o1) | not reported | not reported | agent harness (aider, CLI-centric) | This early-2025 attempt ported one component before being abandoned; the author later attributed the failure to model capability rather than harness sophistication. |
Source for Claude Code and Codex (paired contributor/reviewer) settings
“Between late January and mid-March 2026, two coding agents, Claude Code and Codex, alternated between contributor and reviewer roles to port the entire codebase to PyTorch.”Case Study A, Summary
Source for Claude Code (Sonnet 4.5/4.6) settings
“Claude Code (Sonnet 4.5/4.6) was used throughout.”Case Study B, header
Source for GPT-5.5 Pro (API) and Codex settings
“we used OpenAI GPT-5.5 Pro API and Codex to identify and harden the artifact-prone and bottlenecked steps in BamSurgeon”Case Study D, Summary
Source for GPT-5.5 settings
“the best optimized implementation from GPT-5.5 reduced runtime by 25.1%”Case Study E, Summary
Source for GPT-5.2 settings
“Using GPT-5.2, we rewrote bayesm in Rust as bayesm-rs, achieving feature parity with the original library alongside 2×–20× speedups on reference workloads across different functions.”Case Study G, Summary
Source for GPT-5.2 then GPT-5.6 (successive optimization passes) settings
“GPT-5.2, when prompted in a zero-shot manner to identify potential optimizations to the HI.SIM genomic read simulation library, was able to identify a series of additive, low-level changes resulting in a 23.72% runtime reduction on a reference workload while preserving byte-level output equivalence.”Case Study H, Summary
Source for Claude 3.5 Sonnet and OpenAI o1 (via aider, early abandoned attempt) settings
“One of us (S.F.) first tried to migrate MHCflurry with aider,8 an early open-source CLI-centric agent harness, using Claude 3.5 Sonnet and OpenAI's o1 model.”Case Study A, Obstacles and failure modes
What they reported — and what they left out
The report names which LLM or agent harness was used in each case study (Claude Code, Codex, GPT-5.2, GPT-5.5, GPT-5.6, plus historical Claude 3.5 Sonnet and o1) and sometimes the workflow pattern (e.g., adversarial contributor/reviewer pairing), but gives no temperature, sampling, context-window, or reasoning-effort settings for any of them, and two case studies (C, F) do not name a specific underlying model at all.
The numbers they report
rustar-aligner closely reproduces STAR's alignment behavior on a fixed read set.
99.815% single-end / 99.883% paired-end tie-adjusted parity
See it in the paper
“It reaches 99.815% single-end and 99.883% paired-end tie-adjusted parity with STAR, with no reads mapped by one tool and not the other, and a suffix array byte-for-byte identical to STAR’s.”Case Study B, Summary (rustar-aligner)
RustQC dramatically speeds up RNA-seq quality-control processing while matching the original tools' output.
more than 60x faster (15h34m to 14m54s); disk I/O 2.5TB to 0.1TB
See it in the paper
“RustQC finished the same work in 14 minutes 54 seconds, more than 60x faster, with disk I/O down from 2.5TB to 0.1TB.”Case Study C, Outcome
HelixForge's GPU-native mutation-injection step is far faster than BamSurgeon's CPU pipeline.
98.6x faster (editing step: 1,556.9s to 15.8s)
See it in the paper
“On the editing step, BamSurgeon averaged 1,556.9 seconds, while HelixForge averaged 15.8 seconds for the same task, making the GPU-native path about 98.6x faster.”Case Study D, Validation of the workflow
HelixForge also produces more accurate injected mutation frequencies than BamSurgeon.
mean error 0.076 to 0.034
See it in the paper
“Mutation-frequency accuracy also improved, with the average error falling from 0.076 to 0.034.”Case Study D, Validation of the workflow
Agentic optimization of hifiasm reduced runtime on a held-out synthetic genome-assembly benchmark while meeting a preset quality threshold.
25.1% reduction (816.9s to 612.0s)
See it in the paper
“The strongest result reduced held-out synthetic runtime by approximately 205 seconds, from 816.9 seconds to 612.0 seconds, while preserving the required ordering-quality metrics.”Case Study E, Validation and evidence
The hifiasm optimization's speedup transferred to real human sequencing data, though at a smaller magnitude.
14.7% reduction (734.8s to 626.6s)
See it in the paper
“Runtime decreased from 734.8 seconds to 626.6 seconds, saving 108.2 seconds and producing a 14.7% reduction.”Case Study E, Validation and evidence
The new svb compression library is faster than the most widely used comparable library while remaining wire-compatible with it.
1.7x to 2.9x faster
See it in the paper
“It is wire-compatible with existing StreamVByte data and runs consistently 1.7 to 2.9x faster than streamvbyte64, the most used existing crate.”Case Study B, Summary (svb)
The Rust rewrite of bayesm's core sampler is substantially faster than the original R/C++ package, especially multithreaded.
2.31x (1 thread) to 9.51x (8 threads) at R=200,000 draws
See it in the paper
“the rewrite was 2.31× faster single-threaded and 4.35× faster on eight threads; at R = 200,000, it was 2.71× and 9.51× faster.”Case Study G, Validation and evidence
After correcting a first-pass defect, the bayesm.HART rewrite ended up faster than the original implementation.
3.6x slower before correction; 2.6x faster after correction
See it in the paper
“The corrected rewrite is now about 2.6× faster than the original bayesm.HART implementation.”Case Study G, Corrected validation and evidence
Two successive agentic optimization passes on the HI.SIM read simulator cut aggregate runtime substantially while preserving exact output.
30.97% aggregate reduction (23.72% first pass + 9.5% additional second pass)
See it in the paper
“Two successive zero-shot optimization passes reduced aggregate runtime by 30.97% across a representative four-workload suite while preserving byte-level output equivalence and the simulator’s underlying statistical behavior.”Case Study H, Outcome
The MHCflurry TensorFlow-to-PyTorch migration involved a large, agent-executed code change.
~10,000 lines changed across ~130 files
See it in the paper
“The agents changed nearly ten thousand lines of code across roughly a hundred and thirty files while preserving the ability to load previously trained weights unchanged.”Case Study A, Summary
Published academic research code frequently fails to run, motivating the paper's modernization argument.
74% failed on first run; 56% still failed after automated cleaning (n>9,000 R scripts)
See it in the paper
“In a study of more than 9,000 published R scripts, 74% failed on the first run in a clean environment and 56% still failed after automated cleaning.”Discussion
Omics software installability is similarly poor, per a cited prior study.
57.1% failed documented install instructions; 27.6% still failed after manual intervention
See it in the paper
“among 98 omics tools, 57.1% failed when the documented installation instructions were followed, 27.6% could not be installed after manual intervention, and an automatic failure required about 70 additional minutes of work on average.”Discussion
The authors' illustrative cost model suggests large potential researcher-time savings from agent-assisted modernization.
$6,000-$49,000 per 1,000 attempts; $0.6-4.9M per 100 packages
See it in the paper
“this corresponds to $6,000–$49,000 across those 1,000 attempts, or $0.6–4.9M across 100 packages under the same assumptions.”Discussion
The MHCflurry PyTorch port's predictions closely match the original TensorFlow model's outputs.
315 allele-and-peptide combinations, agreement within a very small error tolerance
See it in the paper
“PyTorch outputs were checked against the TensorFlow backend predictions across 315 allele-and-peptide combinations and agreed to a very small error tolerance across all predicted quantities”Case Study A, Validation and evidence
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
Coding agents in scientific computing hold great promise, but outstanding concerns remain around responsibility and ownership of the resulting software.
“Overall, we find that the use of coding agents in scientific computing holds great promise for accelerating scientific research and increasing the reliability of critical systems, but that outstanding concerns remain, including responsibility and ownership for such projects”
“there is a serious risk that the proliferation of inexpensive rewrites may simply divide users between superficially similar tools.”
Stewardship remains a critical aspectGreater agent autonomy in implementation did not remove the need for human judgment; it shifted human effort toward specification, validation design, and interpretation.
“Across those seven cases, greater agent autonomy in implementation did not remove the need for human judgment; instead, it shifted human effort toward problem specification, validation design, and scientific interpretation.”
“In all but one case, contributors remained the principal adjudicators of success: they defined representative datasets and downstream quantities which served as required success gates”
Recurring ThemesAgent self-assessments do not provide reliable evidence that a task is actually complete.
“contributors reported that while agents could implement many well-scoped tasks, agent selfassessments did not provide reliable evidence of completion.”
“The agent will assert that a plot looks correct, but on visual inspection it is severely flawed: overlapping labels, an incorrect axis, an element flipped or rotated.”
Case Study B, Obstacles and failure modes (kuva)Coding agents were most effective when the intended result could be checked against an external, objective reference.
“Coding agents were most effective when the intended result could be checked against an external reference, such as byte-identical output, posterior agreement with an existing implementation, predictions from released statistical models, or existing test suites.”
“We consistently observed that agents were most effective when their task specifications were narrow and the tests were fully objective.”
Case Study D, Obstacles and lessonsApparently strong aggregate agreement between a rewrite and the original can coexist with important underlying implementation defects.
“The first-pass HART output nevertheless had a 0.991 correlation with the original prediction surface, illustrating how apparently strong aggregate agreement can coexist with important implementation defects.”
“11 of 14 coefficients had cross-respondent mean gaps above 0.25 reference SDs, with the focal Out_State coefficient having a signed difference of −1.20.”
Case Study G, Initial bayesm.HART validationReal-world data is important for determining whether performance improvements found on synthetic workloads actually transfer.
“The use of real-world data was found to be important for determining whether performance improvements transferred beyond small or synthetic workloads.”
“In hifiasm, the runtime reduction was smaller on recorded human reads than on the held-out synthetic benchmark, illustrating that performance gains can attenuate depending on the specific datasets.”
Validation and scientific correctness remain the bottleneckAgent-assisted modernization could plausibly return substantial researcher time and cost savings, illustrated with an order-of-magnitude estimate.
“this corresponds to $6,000–$49,000 across those 1,000 attempts, or $0.6–4.9M across 100 packages under the same assumptions.”
“Under an illustrative scenario in which agent-assisted modernization prevented one quarter to one half of those failures, approximately 69–280 additional analyses would become runnable.”
DiscussionHow they frame it, and what they want next
Their framing
The authors frame their findings with notable epistemic caution for an industry-authored report: they repeatedly caveat that case-study numbers are contributor-reported rather than independently replicated, that their economic estimates are illustrative rather than rigorous, and that the central bottleneck is human validation and stewardship capacity rather than agent capability.
Register: The paper is confident about its qualitative pattern (agents help, but validation is the bottleneck) while being unusually explicit and repeated in flagging that its quantitative claims, both the case-study benchmarks and the economic-value scenarios, are contributor-reported or illustrative rather than independently verified.
Where they hedge
“Unless otherwise stated, numerical results should therefore be interpreted as contributor-reported, case-specific outcomes rather than independently replicated estimates of agent performance.”Case Studies
“While these are very rough estimates, they illustrate the scale of researcher time that could be saved if modernization meaningfully reduced installation and execution failures.”Discussion
“This exploratory field report is retrospective: the underlying projects were not commissioned for this study or conducted under a common protocol”Limitations
“Given that pull requests vary widely in scope, and repository activity does not record the labor required to produce them, these numbers should be treated merely as illustrative.”Ongoing software maintenance
What they say it means
- Coding agents could make tested, optimized software releases a routine expectation for academic code rather than an exception.
the paper’s words
“The advent of coding agents, corresponding with dramatically lower software engineering costs, could make an optimized, tested-first release a routine expectation for academic code releases and allow for rapid modernization of established packages.”Discussion
- The economic opportunity from agentic coding is better understood as reallocating scarce expert effort toward specification, verification, and stewardship, not eliminating the need for that effort.
the paper’s words
“The economic opportunity is therefore better understood as a reallocation of scarce expert effort from implementation toward specification, verification, and stewardship.”Assisting and enabling new package development
- Uncoordinated, cheap rewrites risk fragmenting user communities and diffusing the human oversight needed to make any one rewrite trustworthy.
the paper’s words
“there is a serious risk that this diffusion of attention will result in ecosystems of software rewrites where no one rewrite is actually validated to an extent that permits real-world usage”Stewardship remains a critical aspect
What they call for next
- Begin communicating and coordinating with a tool's existing maintainers well before a rewrite is ready for release.
the paper’s words
“Communication with existing maintainers should therefore begin well before a rewrite is ready for release.”Stewardship remains a critical aspect
- Pursue collaboration and stewardship with existing maintainers when feasible, rather than letting agent-assisted rewrites fork away unsupervised.
the paper’s words
“we suggest collaboration and stewardship with existing maintainers when feasible.”Abstract
Limitations they state
“This exploratory field report is retrospective: the underlying projects were not commissioned for this study or conducted under a common protocol, and the case studies were collected from contributors after the work had already been undertaken.”Limitations
“They therefore provide a narrow, selected cross-sectional view of current practice rather than a representative sample of agent-assisted scientific software projects or developers.”Limitations
“Assessments of human effort, time saved, and economic advantage rely largely on contributors’ qualitative judgments rather than prospectively collected quantitative measurements.”Limitations
Moves worth stealing
Explicitly disclaims responsibility for each case study's own reported numbers, pushing accountability to the named contributor-authors rather than the paper's central authors.
“The authors of each case study are responsible for the accuracy and interpretation of its project-specific claims, including reported benchmarks and validation results.”
Publishes a contributor's own self-deprecating, first-person account of an earlier abandoned attempt rather than smoothing the narrative into unbroken success.
“Why 200 commits? Because I did this almost entirely with aider... I have learned a lot about how incredibly naive AI-code generation is if you just let it tell its own stories without constant questioning, re-questioning and demands for more tests/debugging/logging/analysis.”
Frames its economic-value argument as an explicit order-of-magnitude illustration with wide uncertainty ranges rather than a confident point estimate.
“At a fully loaded labor cost of $75–$150 per hour, this corresponds to $6,000–$49,000 across those 1,000 attempts, or $0.6–4.9M across 100 packages under the same assumptions.”
Where else this leads
Same people
- Introducing LifeSciBench OpenAI
shares Andrew Ho, Suyash Shringarpure
Published alongside it
The nearest publications in time, across all three labs.
- Discovering cryptographic weaknesses with Claude Anthropic
2026-07-28 - Visual prompt engineering for video models Google DeepMind
2026-07-28 - How enabling two settings tripled our scores on the ARC-AGI-3 benchmark OpenAI
2026-07-29 - Ten advances in mathematics and theoretical computer science OpenAI
2026-08-01
What this page was built from
Extracted from the full field report text, including the main paper (Introduction through Conclusion, references, author contributions) and all eight appendix case studies (A-H); the plain-text extraction occasionally merges words that were hyphenated across a line break in the original (e.g., 'highthroughput', 'oneshot', 'selfassessments') and reorders some table cells into column-major rather than row-major order (notably the Table 1 summary grid), so a few table-only figures were sourced from the cleaner prose restating them elsewhere in the case studies instead.