Solipsistic superintelligence is unlikely to be cooperative
Argues capable AI optimized against a fixed world destabilizes once deployed among adaptive humans, institutions, and other AI; cooperation, not more capability, is the real bottleneck.
Names a specific failure mode, self-undermining unilateral optimization, that survives perfect alignment and bears directly on how a frontier lab should design evaluation and deployment pipelines.
Rakshit S Trivedi · Natasha Jaques · Logan Cross · Alexander Sasha Vezhnevets · Joel Z Leibo — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
“AI’s central challenge is shifting from capability to coexistence. The dominant paradigm in AI research focuses on developing powerful agents that treat the world as an exogenous and stationary source of feedback. We contend that superintelligence, an extremely capable task solver, born out of such a solipsistic approach to AI design, is unlikely to be cooperative. Deploying AI systems induces endogenous non-stationarity, resulting in a train–test–deploy gap where historical distributions diverge from the deployment context. We refer to this as the self-undermining property of unilateral optimization. Closing this gap requires AI that participates in cooperation: the equilibrium-selection process through which multiple actors navigate their interdependence. We call for a non-solipsistic research paradigm that treats this interdependence as a core design principle rather than approaching cooperation as a task to solve. This entails building dynamic evaluation testbeds involving adaptive counterparties, treating institutions as design primitives, and preserving human agency as a structural feature of the systems we build.”
The authors argue that mainstream AI development trains and evaluates systems as if the world were a fixed target that never responds, but real deployment puts AI among humans, institutions, and other AI that all adapt to it. That adaptation opens a gap between how a system performed in training and how it performs once deployed, and the more aggressively a system exploits old patterns, the faster that gap can widen. They conclude that building genuinely cooperative AI requires new kinds of adaptive evaluation, treating rules and institutions as things to design, and protecting people's ability to meaningfully participate, rather than just scaling up a single optimizer.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
Solipsistic superintelligence
“By solipsistic superintelligence, we refer to the product of this paradigm pushed to its limit. It represents an extremely capable AI (perhaps one that “solves all stationary tasks”) built on assumptions that held historically up to the point of deployment but no longer hold afterwards.”Introduction
In plain terms: An extremely capable AI built entirely on the assumption that the world it acts in will not react to it.
Cooperation
“negotiation process by which a society coordinates to select beneficial equilibria and avoid harmful ones.”Section 2.1
In plain terms: Cooperation is the ongoing process of picking good shared arrangements, not a fixed end state everyone settles into.
Endogenous non-stationarity
“A learning problem exhibits endogenous non-stationarity when the deployment of policy π induces changes in the transition dynamics P or reward proxy R through response adaptations of other agents.”Section 3.2, Definition 3.1
In plain terms: The world an AI faces changes precisely because the AI itself was deployed into it, not for any outside reason.
Train-test-deploy gap
“The train-test-deploy gap is the divergence between performance evaluated on historical (exogenous) data and performance under deployment (endogenous) conditions”Appendix A.2, Definition A.2
In plain terms: The difference between how well a system did on its training and test data versus how well it actually does once deployed among reacting agents.
Self-undermining property
“We call this the self-undermining property: the more aggressively a unilateral optimizer exploits historical patterns, the faster it induces the adaptations that render those patterns obsolete.”Section 3.2
In plain terms: The harder a system leans on a pattern it found in the past, the faster it causes that pattern to stop being true.
Markov game
“The environment ceases to be exogenous and becomes a Markov game (Shapley, 1953; Littman, 1994), a multi-player game with strategic counterparties whose policies co-evolve with each other.”Section 3.2
In plain terms: A decision-making setting with several strategic players whose choices all shift in response to one another, rather than one player against a fixed backdrop.
Autocurricula
“Such algorithmic evolution produces autocurricula (Leibo et al., 2019), the emergent training distributions generated by the interaction of learning systems that no single system’s designers intended or anticipated.”Section 3.3, Algorithmic adaptation
In plain terms: Training conditions that emerge on their own from multiple learning systems interacting, which nobody explicitly designed.
Equilibrium selection risk
“Equilibrium selection risk is distinct from standard notions of AI risk focused on misalignment or capability. A perfectly aligned system can nonetheless tip a sociotechnical system into an inferior equilibrium through the strategic responses its presence induces, even when no individual action it takes is misaligned.”Appendix A.5, Definition A.8
In plain terms: The risk that deploying a system pushes society into a worse stable arrangement, even if the system itself never does anything misaligned.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Motivate the problem with near-future illustrative scenarios where each AI performs its own task correctly yet the joint outcome fails.
Trace this step to the paper
“As the evening progresses, this results in empty tables in fully-booked restaurants, surge prices for nonexistent availability and hundreds unable to dine.”Section 1, Introduction
- Distinguish the paper's target problem (cooperation) from the standard alignment framing, arguing perfect specification-following is not sufficient.
Trace this step to the paper
“A system can be perfectly aligned with its specification, values included, and still make things worse once it acts among other adaptive systems.”Section 2.1
- Formalize the shift from single-agent MDPs to multi-agent Markov games and define endogenous non-stationarity.
Trace this step to the paper
“Definition 3.1. A learning problem exhibits endogenous non-stationarity when the deployment of policy π induces changes in the transition dynamics P or reward proxy R through response adaptations of other agents.”Section 3.2
- Identify and empirically document three channels through which deployment induces adaptation: behavioral, institutional, and algorithmic.
Trace this step to the paper
“The train-test-deploy gap arises through three channels, each representing a distinct class of best-responding agents.”Section 3.3
- Argue that modeling other agents' adaptations cannot rescue unilateral prediction, citing epistemic limits (novelty, reflexivity, combinatorial explosion).
Trace this step to the paper
“This section argues that the objection fails on two independent grounds either of which may suffice to block unilateral prediction as a solution path.”Section 4
- Argue that legitimacy constraints independently block unilateral optimization even if prediction of other agents' behavior were possible.
Trace this step to the paper
“Unilateral optimization would still face another barrier: the legitimacy constraints that open societies impose on prediction, steering, and control (Habermas, 1975; Rawls, 1993; Pasquale, 2015; Crawford & Schultz, 2014; Hadfield & Weingast, 2014).”Section 4.2
- Formalize dynamic evaluation as a (D, µ) tuple and propose a non-solipsistic research agenda spanning dynamic evaluation, institutions as design primitives, and preserving human agency.
Trace this step to the paper
“We formalize an evaluation procedure as a tuple (D, µ), where D is a test distribution over interaction trajectories and µ is a scoring functional mapping the AI’s behavior under D to a real-valued score.”Section 5.1
- Preempt anticipated objections directly with a structured argument-and-rebuttal section.
Trace this step to the paper
“Here we discuss the central alternative views our exposition invites. Appendix D additionally summarizes rebuttals to an extended set of objections.”Section 6
Exactly what was run, and how
What they reported — and what they left out
This is a theoretical position paper: it cites other groups' models and empirical studies (e.g., LLM pricing agents, alignment-faking work) but does not itself run, fine-tune, or evaluate any AI model, and reports no model name, temperature, or deployment settings of its own.
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
Even a system that perfectly follows its own specification can still make collective outcomes worse once it acts among other adaptive systems.
“A system can be perfectly aligned with its specification, values included, and still make things worse once it acts among other adaptive systems.”
“Recommendation algorithms optimized for engagement produce polarization as a byproduct of success (Germano et al., 2026; Milli et al., 2025), pricing algorithms interacting in markets learn supracompetitive prices without explicit communication (Calvano et al., 2020), and automated order flow and liquidity provision interact to produce the kind of instability seen in the Flash Crash (Kirilenko et al., 2017).”
Section 2.1Cooperation is a necessary condition for sustained beneficial outcomes, not an optional extra capability to add on top of a scaled-up optimizer.
“In a world with many humans and many AIs, cooperation is neither optional nor an additional capability to be scaled but a necessary condition for sustained beneficial outcomes.”
“On a Friday evening in 2027, three competing AI reservation systems in San Francisco calculate optimal release times and learn to make phantom bookings so as to maximize confirmed seats for their users.”
Section 1, IntroductionCompetitive and evolutionary selection pressure will not reliably produce cooperative AI on its own, because AI deployment violates the conditions under which selection favors cooperation.
“Selection does produce cooperation, under specific conditions that AI deployment systematically violates.”
“LLM-based pricing agents autonomously converge on supracompetitive prices in oligopoly settings (Fish et al., 2026), divide markets in multi-commodity Cournot competition (Lin et al., 2025), and self-play Q-learners provably learn collusive policies in iterated social dilemmas (Bertrand et al., 2025).”
Section 7, ConclusionThe absence of catastrophic multi-agent AI failures so far does not mean the solipsistic approach is adequate.
“So the claim that the empirical track record does not support alarm is itself highly questionable.”
“Recommenders have not collapsed society, but they have measurably increased polarization, degraded epistemic commons, and reshaped political discourse in ways that democracies are struggling to absorb (Germano et al., 2026; Milli et al., 2025).”
Section 6, Rebuttal to Argument 3More aggressive exploitation of historical patterns by a policy accelerates the very adaptations that make those patterns stop working.
“the more aggressively a unilateral optimizer exploits historical patterns, the faster it induces the adaptations that render those patterns obsolete”
“This occurs when the policy exploits patterns that depend on other actors’ current strategies. As the policy extracts more value from these patterns, it strengthens incentives for other actors to adapt by changing strategies, seeking alternatives, or exiting the interaction entirely.”
Appendix A.3Training a model to behave cooperatively during RLHF or Constitutional AI does not establish that it will behave cooperatively once deployed.
“RLHF and Constitutional AI train models to exhibit helpful, harmless, and honest behavior against a fixed distribution of human feedback (Bai et al., 2022), but deployment changes the game.”
“The training signal itself becomes a target, with sufficiently capable models learning to present as cooperative during evaluation while pursuing divergent objectives when conditions change”
Appendix D, Rebuttal to Argument 5How they frame it, and what they want next
Their framing
The authors frame their argument as a structural corrective to mainstream AI research rather than an alarmist warning: they credit the alignment literature's insights, then argue a categorically different failure mode, self-undermining unilateral optimization, is being treated as an edge case rather than a central design constraint. They use boxed 'Key Claim' statements to anchor the argument and dedicate whole sections to preempting six anticipated objections rather than leaving them to reviewers.
Register: Confidence reads high and assertive at the level of the thesis, stated flatly in the title and in boxed Key Claims, but the authors insert explicit hedges wherever a claim rests on an unresolved empirical counterfactual, such as why catastrophic multi-agent failures have not yet been observed at today's deployment scale.
Where they hedge
“While we expect solipsistic methods to remain effective in narrow domains, our paper targets sociotechnical settings where advanced AI deployment will be heavily exposed to response dynamics (“push back”).”Scope paragraph, Introduction
“The absence of catastrophe thus far may reflect the limited capability and deployment scale of current systems or the short time period for human adaptation rather than the adequacy of the solipsistic approach.”Section 6, Rebuttal to Argument 3
“Multiple different equilibria are generally possible, and they may differ sharply in welfare and distributional consequences.”Section 2
What they say it means
- Standard offline train/test evaluation cannot certify deployment performance once endogenous non-stationarity is present.
the paper’s words
“Offline evaluation is insufficient. Standard train-test splits assume exogenous distributions. Under endogenous nonstationarity (Definition A.1), test performance does not predict deployment performance. Evaluation must incorporate adaptive counterparties.”Appendix A.6
- Making a system more capable on historical benchmarks can make its deployment behavior worse rather than better.
the paper’s words
“Capability improvements may be counterproductive. The self-undermining property (Definition A.3) implies that stronger policies can yield worse deployment outcomes. Optimization pressure on historical benchmarks may select for systems that destabilize upon deployment.”Appendix A.6
- Designers should evaluate which equilibria a policy makes reachable for the whole system, not only what the policy does when other agents are held fixed.
the paper’s words
“Equilibrium welfare is the relevant objective. A policy that is locally optimal can participate in globally suboptimal equilibria. Design must consider what equilibria the policy makes reachable, rather than only what the policy does in isolation (Definition A.8).”Appendix A.6
- Deploying a powerful optimizer should be treated as an intervention that changes the strategic game itself, which governance must account for.
the paper’s words
“Deployment is an intervention. Introducing a powerful optimizer changes the game rather than playing within fixed rules. Design and governance must account for the system’s effect on the strategic environment it enters.”Appendix A.6
What they call for next
- Adopt a non-solipsistic research paradigm that treats multi-actor interdependence as a core design principle rather than a task to solve after the fact.
the paper’s words
“We call for a non-solipsistic research paradigm that treats this interdependence as a core design principle rather than approaching cooperation as a task to solve.”Abstract
- Build dynamic evaluation testbeds, treat institutions as design primitives, and preserve human agency as structural features of the systems being built.
the paper’s words
“This entails building dynamic evaluation testbeds involving adaptive counterparties, treating institutions as design primitives, and preserving human agency as a structural feature of the systems we build.”Abstract
- Make the impact of AI deployment on human skills, autonomy, and meaningful choice a core part of evaluation pipelines, not a separate ethics add-on.
the paper’s words
“it is imperative to include the impact assessment of AI deployment on human skills, autonomy, and meaningful choice as a core part of evaluation pipelines, rather than as a separate ethical concern (Zhuang et al., 2025; Haupt & Brynjolfsson, 2025; Kulveit et al., 2025b).”Section 5.3, Preserving Human Agency
Limitations they state
“Our paper sets aside these definitions pertaining to capability thresholds and instead focuses on the the methodological assumptions of environmental exogeneity, objective stationarity, and singleton framing.”Scope paragraph, Introduction
“While today’s systems already exhibit the dynamics we describe, at the level of superintelligence, our position contests an implicit bet in the dominant methodology that scaling capability will eventually deliver cooperative outcomes the way it has delivered gains in reasoning or coding.”Scope paragraph, Introduction
Moves worth stealing
Uses boxed, numbered 'Key Claim' callouts to make the paper's load-bearing propositions scannable independent of the surrounding prose.
“In a world with many humans and many AIs, cooperation is neither optional nor an additional capability to be scaled but a necessary condition for sustained beneficial outcomes.”
Preempts the strongest anticipated objections in a dedicated 'Alternative Views' section with named arguments and point-by-point rebuttals, rather than leaving them to reviewers.
“Here we discuss the central alternative views our exposition invites. Appendix D additionally summarizes rebuttals to an extended set of objections.”
Opens with a vivid, concrete near-future narrative scenario before any formal claim, to make an abstract multi-agent argument feel immediately consequential.
“On a Friday evening in 2027, three competing AI reservation systems in San Francisco calculate optimal release times and learn to make phantom bookings so as to maximize confirmed seats for their users.”
Explicitly narrows scope in a labeled 'Scope' paragraph early on, pre-empting a category of objection rather than letting readers assume the broadest possible reading.
“Our paper sets aside these definitions pertaining to capability thresholds and instead focuses on the the methodological assumptions of environmental exogeneity, objective stationarity, and singleton framing.”
Backs a purely conceptual claim with a formal appendix of definitions and a proof sketch, giving the informal argument a checkable mathematical skeleton.
“This section provides formal foundations for the concepts introduced in Section 3.”
Where else this leads
Same territory
- Patterns and problems in emerging multiagent systems Anthropic
ai safety alignment - Realistic honeypot evaluations for scheming propensity Google DeepMind
ai safety alignment - SLEIGHT-Bench: Finding Blind Spots in AI Monitors Anthropic
ai safety alignment - Introducing the Conceptual Reasoning Index Anthropic
ai safety - TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic
ai safety - Training a Misaligned Reward Seeker Anthropic
ai safety
Published alongside it
The nearest publications in time, across all three labs.
- Gram: Assessing sabotage propensities via automated alignment auditing Google DeepMind
2026-05-28 - Realistic honeypot evaluations for scheming propensity Google DeepMind
2026-05-28 - From AGI to ASI Google DeepMind
2026-06-12 - Diffuse AI Control on Fuzzy Tasks Anthropic
2026-06-15
What this page was built from
Working from the full extracted text of the ICML paper (title, abstract, all numbered sections 1-7, and Appendices A-D with references); the source PDF's two-column layout appears to have interleaved a few lines out of order near the top (e.g., a stray sentence appearing just before the actual Abstract paragraph) and rendered inline math with minor spacing artifacts.