Would This Change Your Answer? Evaluating Explanations of LLM Behavior in the Wild with Counterfactual Experiments
CHIVE discovers unexpected LLM behaviors and explains them via counterfactual edits; interpretability tools gave no predictive uplift, but models trained on this data generalized to new settings.
It's evidence that today's activation-reading interpretability tools may not yet earn their keep at predicting model behavior, which matters for anyone deciding how much to trust such tools in safety evaluations.
Adam Karvonen · Euan Ong · Subhash Kantamneni · Samuel Marks — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
This source carries no verbatim abstract.
The authors built CHIVE, a pipeline that automatically finds surprising behaviors in a language model's real outputs and tests explanations for them by editing the prompt and re-running the model. Using this data, they found that giving an AI predictor access to interpretability tools that read the model's internal activations did not help it guess how a prompt edit would change the behavior, compared to just reading the transcript. The same data, when used to train models to predict their own behavior under prompt edits, did generalize well to new, untrained-on settings.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
Counterfactual simulatability
“we evaluate explanations through the lens of counterfactual simulatability : a good explanation of a behavior should help you predict what the model will do on related counterfactual inputs.”Introduction
In plain terms: A good explanation of why a model did something should let you predict what it would do if you changed the situation slightly.
CHIVE (Counterfactual Hypothesis Investigation Via Edits)
“We introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), an agentic pipeline that generates such data automatically.”Introduction
In plain terms: An automated pipeline that finds odd model behaviors and tests explanations for them by editing prompts and re-running the model.
Activation oracle (AO)
“Activation oracles (AOs): models trained to answer arbitrary natural-language questions about activations.”Interpretability tools provide no uplift on our evaluation
In plain terms: A tool you can ask free-form questions about what's happening inside the model at a given moment.
Natural-language autoencoder (NLA)
“Natural-language autoencoders (NLAs): models trained to produce an open-ended description of a given activation.”Interpretability tools provide no uplift on our evaluation
In plain terms: A tool that describes, in plain language, what a snapshot of the model's internal activity seems to represent.
Sparse autoencoder (SAE)
“Sparse autoencoders (SAEs): dictionaries that decompose an activation into sparse features, each with a natural-language description.”Interpretability tools provide no uplift on our evaluation
In plain terms: A tool that breaks an internal activation down into a small set of labeled, individually-understandable pieces.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Sampled the target model on a researcher-chosen set of prompts, generating many responses per prompt.
Trace this step to the paper
“Sample. Run the target model on a researcher-specified set of prompts, sampling 30 responses per prompt.”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
- Had an investigator model read the sampled responses and flag behaviors that looked unexpected.
Trace this step to the paper
“Screen. An investigator model reads the responses and flags unexpected behaviors.”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
- Had an investigator agent run a batch of counterfactual prompt-edit experiments per flagged behavior, re-sampling the model each time and measuring how the behavior's frequency changed.
Trace this step to the paper
“Investigate. An investigator agent runs 5–15 counterfactual experiments to explain what drives each behavior. Each experiment edits the prompt, resamples the target model, and measures the change in how often the behavior occurs.”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
- Had an independent judge model review the experiment set and score how well it supported the proposed explanation.
Trace this step to the paper
“Verify. An independent judge reviews the experiments and scores how well they support the explanation.”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
- Built a prediction task where a predictor agent sees a transcript, a described behavior, and one candidate counterfactual, and must judge whether that edit would change the behavior.
Trace this step to the paper
“Each predictor agent (Claude Opus 4.8 in our main experiments) receives a transcript, a behavior, and one proposed counterfactual, and outputs the probability that the counterfactual would change the behavior.”Interpretability tools provide no uplift on our evaluation
- Gave some predictor agents a limited number of read-only calls into the target model's activations, using one of three interpretability tools previously shown useful in other auditing settings.
Trace this step to the paper
“Tool predictors can additionally make 5 read-only calls on the target model's activations, using one of three tools, each chosen because it provided uplift in prior auditing games on fine-tuned models”Interpretability tools provide no uplift on our evaluation
- Repurposed the same investigation data as training examples, framed as follow-up turns on the model's own transcript with a single counterfactual claim, to train models to predict their own behavior under prompt edits.
Trace this step to the paper
“We train models to predict the outcomes of counterfactual prompts. Each training example is a follow-up turn on the model's own transcript with a single claim (Figure 3, left).”Training models to predict their own behavior
- Trained two target models on this data and tested whether the trained behavior-prediction skill generalized to a held-out setting (the hint setting) never targeted during training.
Trace this step to the paper
“We train two target models, Qwen3-8B and Qwen3.5-397B-A17B.”Training models to predict their own behavior
Exactly what was run, and how
| Model | Developer | Temp | Effort / reasoning | Deployment | Other settings |
|---|---|---|---|---|---|
| Claude Opus 4.8 | Anthropic | not reported | not reported | unstated | Used as the main predictor agent in the interpretability-tool evaluation, receiving a transcript plus up to 5 read-only activation-tool calls; also used as an external reference model reading the same transcripts in the hint-setting counterfactual-prediction evaluation. |
| Qwen3-8B | — | not reported | not reported | fine-tuned | One of two target models trained on CHIVE-derived data to predict the outcomes of counterfactual prompt edits on its own behavior. |
| Qwen3.5-397B-A17B | — | not reported | not reported | fine-tuned | One of two target models trained on CHIVE-derived data to predict the outcomes of counterfactual prompt edits on its own behavior. |
| Gemma | — | not reported | not reported | unstated | Referenced as the target model in the paper's Figure 1 illustrative example (a coding-error behavior traced to misleading parameter names); not otherwise described as part of the main tool-evaluation or training experiments. |
Source for Claude Opus 4.8 settings
“Each predictor agent (Claude Opus 4.8 in our main experiments) receives a transcript, a behavior, and one proposed counterfactual, and outputs the probability that the counterfactual would change the behavior.”Interpretability tools provide no uplift on our evaluation
Source for Qwen3-8B settings
“We train two target models, Qwen3-8B and Qwen3.5-397B-A17B.”Training models to predict their own behavior
Source for Qwen3.5-397B-A17B settings
“We train two target models, Qwen3-8B and Qwen3.5-397B-A17B.”Training models to predict their own behavior
Source for Gemma settings
“the explanation "Gemma makes this coding error because it’s misled by the parameter names" (Figure 1) predicts that renaming the parameters should prevent the error.”Introduction
What they reported — and what they left out
The paper names Claude Opus 4.8 as the main predictor and reference model, Qwen3-8B and Qwen3.5-397B-A17B as the trained target models, and Gemma as the target model in one illustrative example, but it never reports temperature, sampling parameters, context length, or deployment mode (API vs. web UI) for any of them, and states only a bare count of tool calls ('5 read-only calls') without further configuration detail.
The numbers they report
None of the three activation-reading interpretability tools outperformed the transcript-only baseline predictor.
See it in the paper
“None of the three tools beats the transcript-only baseline (Figure 4).”Interpretability tools provide no uplift on our evaluation
The no-uplift result was robust across multiple target models, predictor model families, and hyperparameter settings.
2 target models; 3 predictor model families (exact hyperparameter sweep counts not given)
See it in the paper
“The result holds across many variations, including two target models, three predictor model families, sweeps of hyperparameters, and manual and automated attempts to elicit better tool use.”Interpretability tools provide no uplift on our evaluation
Each behavior investigation involved a bounded number of counterfactual experiments.
5-15 experiments per investigation
See it in the paper
“An investigator agent runs 5–15 counterfactual experiments to explain what drives each behavior.”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
The pipeline samples a fixed number of responses per prompt to detect unexpected behaviors.
30 responses per prompt
See it in the paper
“Run the target model on a researcher-specified set of prompts, sampling 30 responses per prompt.”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
Tool-equipped predictors were limited to a small, fixed number of activation queries.
5 read-only calls
See it in the paper
“Tool predictors can additionally make 5 read-only calls on the target model's activations”Interpretability tools provide no uplift on our evaluation
Models trained on CHIVE data substantially improved at predicting their own behavior in the held-out hint setting.
See it in the paper
“Each trained model improves substantially over its base model (Figure 5).”Training models to predict their own behavior
The trained self-prediction skill generalized beyond the hint setting to other held-out investigations, including from an out-of-distribution transcript source.
See it in the paper
“It also generalizes to held-out investigations from the pipeline, including ones built from an out-of-distribution source of transcripts.”Training models to predict their own behavior
Training models to produce open-ended self-explanations (rather than yes/no counterfactual predictions) worked less well.
See it in the paper
“We also experimented with training models to generate open-ended explanations of their own behavior (Figure 3, right), with weaker mixed results; see our paper's appendix for details.”Training models to predict their own behavior
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
Activation-reading interpretability tools provide no predictive advantage over simply reading the transcript.
“Surprisingly, no predictor outperforms one that is just shown the transcript with no access to interpretability tools.”
“None of the three tools beats the transcript-only baseline (Figure 4). The result holds across many variations, including two target models, three predictor model families, sweeps of hyperparameters, and manual and automated attempts to elicit better tool use.”
Interpretability tools provide no uplift on our evaluationThe negative result isn't simply because predictor agents ignore the interpretability tools outright.
“Our negative result is not because the predictor always ignores the tools.”
“on the randomNum behavior from Figure 1, the NLA reads out that the model is "likely generating a random integer between min and max", which helps the agent predict that the error is due to the misleading parameter names. But these wins are offset by cases where the tool output misses the relevant information or actively misleads the predictor.”
Why don't the tools help?Applied claims about a model's internal beliefs (e.g., that it viewed an action as legitimate) implicitly rest on a counterfactual that this evaluation's results speak to.
“But these claims often make an implicit counterfactual suggestion.”
“Our results suggest that interpretability tools that read activations provide limited evidence for these causal claims.”
How should we interpret these results?Training models on CHIVE-derived counterfactual data produces a self-prediction skill that generalizes beyond its training distribution.
“Training generalizes to the hint setting, which was not targeted during training.”
“Each trained model improves substantially over its base model (Figure 5). It also generalizes to held-out investigations from the pipeline, including ones built from an out-of-distribution source of transcripts.”
Training models to predict their own behaviorInterpretability tools remain valuable despite showing no measurable uplift in this evaluation.
“Overall, we still believe these tools can be valuable, as they provide evidence about internal states that no other method can obtain.”
“Our evaluation is a close checkable proxy, and the tools provided no uplift.”
How should we interpret these results?How they frame it, and what they want next
Their framing
The authors present the negative interpretability-tool result candidly as a genuine puzzle ('Why don't the tools help?') rather than downplaying it, then devote a full section to working through why their evaluation might not transfer to Anthropic's actual system-card use cases (simpler behaviors, sampling-based ground truth, retrospective belief claims) before landing on a bounded conclusion that the tools remain valuable but should currently be treated as only suggestive.
Register: The authors state their central negative finding plainly and back it with an explicit robustness claim, but qualify nearly every interpretive extension of that finding with hedges ('may', 'often', 'we generally cannot tell', 'suggestive evidence'), and devote a dedicated section to bounding exactly how far the result should be read.
Where they hedge
“These are often compelling, but as they are LLM-generated, many are likely omitting important details or partially wrong, so we don't treat them as ground truth.”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
“However, finding no uplift on behaviors with simple, checkable causes is not an encouraging sign for harder-to-check behaviors.”How should we interpret these results?
“Without this comparison, we generally cannot tell how much additional evidence came from our interpretability tools (although corroboration can itself be valuable).”How should we interpret these results?
“Until that changes, we think causal claims based on tool outputs should only be treated as suggestive evidence.”How should we interpret these results?
What they say it means
- Given that activation-reading tools showed no measurable uplift on checkable, in-the-wild behaviors, causal claims made with the same tools about harder-to-check applied cases (such as system-card behavioral claims) warrant corresponding caution.
the paper’s words
“Until that changes, we think causal claims based on tool outputs should only be treated as suggestive evidence.”How should we interpret these results?
- Training self-prediction on thousands of diverse, naturally occurring behaviors (rather than one narrow planted-cue task) can produce a skill that generalizes, suggesting broader training data may be key to generalizable self-prediction.
the paper’s words
“Our training data instead covers thousands of behaviors appearing in the wild with diverse causes.”Training models to predict their own behavior
What they call for next
- Directs readers to the full paper for additional methodological details and results.
the paper’s words
“Read our paper for additional details and results.”In summary
- Invites readers to inspect the underlying investigations directly, including a random sample beyond the hand-picked examples shown.
the paper’s words
“The full investigations of these four are viewable here , and 20 randomly selected investigations here .”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
Limitations they state
“These are often compelling, but as they are LLM-generated, many are likely omitting important details or partially wrong, so we don't treat them as ground truth.”CHIVE: a pipeline for discovering counterfactual explanations for model behaviors
“Our evaluation is a proxy. We obtain our ground truth by sampling the model, which means anyone with sampling access could access the ground truth by just running the counterfactual. In contrast, interpretability is typically used in the system card in cases where no ground truth is available. This creates a distribution shift between our evaluation and our practical use cases.”How should we interpret these results?
“Most system card case studies do not include a transcript-reading reference.”How should we interpret these results?
“We also experimented with training models to generate open-ended explanations of their own behavior (Figure 3, right), with weaker mixed results; see our paper's appendix for details.”Training models to predict their own behavior
Moves worth stealing
Opens with a concrete worked example (a Gemma coding error traced to misleading parameter names) before defining any terminology, grounding an abstract methodological idea in something readable.
“the explanation "Gemma makes this coding error because it’s misled by the parameter names" (Figure 1) predicts that renaming the parameters should prevent the error.”
States a surprising negative result plainly in the tl;dr rather than burying it, then immediately gives it its own dedicated 'why' section instead of moving on.
“Surprisingly, no predictor outperforms one that is just shown the transcript with no access to interpretability tools.”
Turns part of the discussion into an explicit self-audit of the evaluation's own limits relative to the lab's real deployed use case, naming concrete differences rather than asserting general validity.
“This creates a distribution shift between our evaluation and our practical use cases.”
Names each pipeline stage with a single bolded verb and a one-line definition, giving a multi-step agentic pipeline an easily memorizable structure.
“Screen. An investigator model reads the responses and flags unexpected behaviors.”
Where else this leads
Same people
- Verbalizable Representations Form a Global Workspace in Language Models Anthropic
shares Euan Ong, Subhash Kantamneni - Model Spec Midtraining: Improving How Alignment Training Generalizes Anthropic
shares Samuel Marks - Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations Anthropic
shares Subhash Kantamneni, Euan Ong, Samuel Marks - Teaching Claude Why Anthropic
shares Samuel Marks
Same territory
- Introducing the Conceptual Reasoning Index Anthropic
ai safety - TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic
ai safety - Training a Misaligned Reward Seeker Anthropic
ai safety - Patterns and problems in emerging multiagent systems Anthropic
ai safety - Agentic Misalignment in Summer 2026 Anthropic
ai safety - Verbalizable Representations Form a Global Workspace in Language Models Anthropic
interpretability
Published alongside it
The nearest publications in time, across all three labs.
- Automated Researchers Can Mitigate Well-Characterized Alignment Failures Anthropic
2026-08-15 - Characterizing interference weights in a tiny language model Anthropic
2026-08-15 - Fine-Tuned Lie Detectors Failed to Generalize Anthropic
2026-08-15 - Introducing the Conceptual Reasoning Index Anthropic
2026-08-15
What this page was built from
Working from a saved plain-text copy of this Anthropic Alignment Science Blog post (~13,500 characters, graded 'partial' in the corpus manifest); the post links out to a separate full paper and code repository ('Paper', 'Code') and to specific investigation viewers, but only the link labels were captured in the extracted text, not their URLs, so pdf_or_paper_url falls back to the blog post's own URL; the post is dated August 21, 2026 in its byline, more precise than the manifest's month-precision 2026-08-15.