Characterizing interference weights in a tiny language model
Anthropic decomposed a tiny language model into millions of weighted connections and found the most impactful ones are almost always helpful, but pruning still leaves too many to fully interpret.
It gives the first concrete, trained-model evidence that reading a network's raw weights can mislead, and tests a two-metric method (effectiveness, helpfulness) for telling real circuits from harmless-looking noise.
Nicholas L. Turner · Jeffrey Wu · Joshua Batson — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
This source carries no verbatim abstract.
Anthropic trained a very small one-layer transformer and re-expressed its weights as direct 'virtual weight' connections between tokens, positions, features, and output logits. They scored every connection on two axes: how much it actually moves the model's predictions ('effectiveness'), and whether it makes the model's loss better or worse ('helpfulness'). Large-magnitude weights are not reliably useful; the most effective weights are almost always helpful, but even after aggressive pruning, tens of millions of weights remain, so the authors conclude the real bottleneck is finding a better basis for the decomposition, not a better pruning rule.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
interference weights
“linear interactions of interpretable model components through the low-dimensional residual stream which are either irrelevant or harmful to the model's behavior.”Introduction
In plain terms: A weight connection between two model components that looks meaningful mathematically but doesn't actually help the model's predictions (or actively hurts them).
virtual weights
“the model's weights can be re-expressed as global effects between these components called virtual weights”Introduction
In plain terms: The effective strength of the connection between two interpretable components (like a token and an output word) once you multiply through all the model's actual weight matrices.
effectiveness
“We define effectiveness as the magnitude of a weight's effect on the function the model computes. We measure it as a second-order estimate of the KL divergence between the model's outputs with and without the weight (using the Fisher metric ).”Effectiveness and helpfulness
In plain terms: How much removing a given weight would change what the model actually predicts, regardless of whether that change would be good or bad.
helpfulness
“For that we measure helpfulness: the average change in loss when the weight is ablated from the forward pass. If the loss rises when we remove a weight, the weight is helping the model toward the right answer; if the loss falls, it is harmful.”Effectiveness and helpfulness
In plain terms: Whether deleting a weight makes the model's predictions better or worse, measured directly as the change in training loss.
helpfulness mass
“If we instead try to preserve 90% of the sum of sampled positive helpfulness values (a kind of "helpfulness mass")”The model is still dense in this basis
In plain terms: The total amount of loss-improvement a group of helpful weights contributes together, rather than just the count of how many weights are helpful.
region of practical equivalence (ROPE)
“Here we classify each sampled weight against a region of practical equivalence (ROPE) : an interval … of mean-helpfulness values we are prepared to treat as practically zero.”Appendix / ROPE plots
In plain terms: A statistical rule that treats a very small measured effect as effectively zero, so tiny-but-technically-nonzero results aren't mistaken for real findings.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Train a tiny one-layer decoder-only transformer from scratch on English and code text.
Trace this step to the paper
“The transformer we study is a one-layer, decoder-only transformer with residual width d_m = 256, 4 attention heads of dimension … = 64, a width-1024 MLP with ReLU activation, a 4,096-token vocabulary, and a 1,024-token context window, totaling ≈2.9M parameters (≈0.79M excluding the embedding and unembedding matrices).”Appendix / Training details / Transformer
- Train a sparse single-layer transcoder to re-express the MLP's computation in more monosemantic (less overlapping) features.
Trace this step to the paper
“A single-layer transcoder (SLT) was trained on the transformer's activations: it reads the pre-MLP residual-stream activation and is trained to predict the MLP output.”Appendix / Training details / Transcoder
- Decompose the whole trained model into a 'virtual weight' (VW) model: six families of direct weights linking tokens, positions, transcoder features, and output logits.
Trace this step to the paper
“Every connection in the VW model sits within one of six families, each a product of matrices that contracts away the residual dimension ( d_m = 256), shown below.”The virtual weight model
- Compute an 'effectiveness' score for every virtual weight, using a second-order (Fisher-information) estimate of how much it shifts the model's output distribution, averaged over hundreds of millions of training tokens.
Trace this step to the paper
“We estimate this value for every virtual weight in the model over ~537M tokens of the training corpus.”Effectiveness and helpfulness
- Compute a 'helpfulness' score for weights (closed-form for weights that target logits directly; via sampling with confidence intervals for the rest) as the average change in loss when the weight is removed.
Trace this step to the paper
“We compute the average helpfulness for the subsets of weights in our worked examples over the entire training set and attach 95% (Gaussian) confidence intervals to each one. We ran similar measurements for a random sample of the whole population over 1B tokens.”Effectiveness and helpfulness
- Walk through four worked qualitative examples (an ACETYLCHOLINE token prediction, a Chinese-script feature, a French-word-ending feature, and a newline-detecting feature) comparing raw virtual weight, effectiveness, and helpfulness rankings.
Trace this step to the paper
“Having defined effectiveness and helpfulness, we can now use them to identify interference weights and interpret parts of the VW model. We work through four examples.”Effectiveness surfaces helpful weights but does not isolate them
- Sample over a thousand weights spread across the effectiveness distribution and estimate each one's mean helpfulness with a confidence interval, to see how effectiveness and helpfulness relate across the whole model.
Trace this step to the paper
“We first sample 1,111 weights from the six virtual weight families and estimate each one's mean helpfulness. We compute this estimate over 1B tokens, along with a 95% confidence interval.”The effectiveness tail is helpful
- Prune the VW model by removing weights in increasing order of effectiveness and measure the resulting loss on a held-out test set at each sparsity level.
Trace this step to the paper
“First, we remove weights from the VW model in order from least to most effective (moving from right to left above) and measure the pruned model's loss on a held-out test set.”The model is still dense in this basis
- Randomly sample thousands of weights and estimate what fraction are statistically significantly helpful versus harmful, to floor-estimate how sparse a helpfulness-based filter could ever get.
Trace this step to the paper
“Even within our single-layer transformer it is expensive to compute helpfulness for every weight, so instead we take a random sample of 7765 weights and attempt to estimate the fraction of helpful weights to act as an approximate floor on the density this basis can reach.”Weight filters in this basis won't yield much sparser models
Exactly what was run, and how
| Model | Developer | Temp | Effort / reasoning | Deployment | Other settings |
|---|---|---|---|---|---|
| Custom 1-layer decoder-only transformer (~2.9M parameters) | Anthropic (trained for this study) | not reported | not reported | local weights | Adam optimizer (beta = 0.9, 0.95), no weight decay, bfloat16 autocast, gradients globally clipped to 1.5x an exponential moving average of recent gradient norms; trained for 11,933 steps on ~9.8x10^7 unique tokens (95,464 unique sequences) from the Common Corpus (English and code), no data repetition. |
Source for Custom 1-layer decoder-only transformer (~2.9M parameters) settings
“Training used Adam with β = (0.9, 0.95) and no weight decay, under bfloat16 autocast, with gradients globally clipped to 1.5× an exponential moving average (decay 0.95) of recent gradient norms.”Appendix / Training details / Transformer
What they reported — and what they left out
The paper gives exhaustive architecture and training hyperparameters (Appendix) for its own custom 1-layer transformer and transcoder, but this is a from-scratch research model, not a deployed frontier model, so there is no temperature/reasoning-effort setting, no benchmark evaluation, and no hardware or wall-clock training time reported.
The numbers they report
Removing the 70% least-effective virtual weights barely increases loss.
0.01 nats loss increase at 70% sparsity (30% density)
See it in the paper
“This costs approximately 0.01 nats at 70% sparsity (30% density) and under 0.1 nats at 85% sparsity (15% density).”The model is still dense in this basis
Removing 85% of virtual weights costs more but is still modest.
under 0.1 nats loss increase at 85% sparsity (15% density)
See it in the paper
“This costs approximately 0.01 nats at 70% sparsity (30% density) and under 0.1 nats at 85% sparsity (15% density).”The model is still dense in this basis
At the sparsity level where parameter count matches the original transformer, performance drops sharply.
1% density = significantly compromised performance
See it in the paper
“At 1%, where the parameter count roughly matches the original transformer, the pruned model's performance is significantly compromised.”The model is still dense in this basis
Fully expanding the model into virtual weights inflates the parameter count by roughly 100x.
2.9M parameters -> ~331M virtual weights (about 100x)
See it in the paper
“Materializing all six families inflates the parameter count from 2.9M to roughly 331M, about 100×, since each pair of endpoints now carries its own explicit weight rather than sharing the residual stream.”The virtual weight model
Roughly half of sampled weights have a statistically positive (helpful) effect on loss.
47.6% positive mean helpfulness; 12.7% "dead" (no measurable effect)
See it in the paper
“Across the sample, roughly half (47.6%) of all weights have positive mean helpfulness (with 12.7% dead), and even counting only those whose confidence interval excludes zero leaves tens of percent (see more details in the Appendix ).”The model is still dense in this basis
Capturing most of the total 'helpful' contribution needs relatively few weights, but capturing nearly all of it needs far more.
2.43% density for 90% of helpfulness mass; 13.6% density for 99% of helpfulness mass
See it in the paper
“If we instead try to preserve 90% of the sum of sampled positive helpfulness values (a kind of "helpfulness mass"), we estimate that we'd only need 2.43% density ( Appendix ), but this fraction quickly climbs to a similar regime (13.6%) when accounting for 99% of the helpfulness mass.”The model is still dense in this basis
The transformer's training loss dropped substantially but had not fully converged when training stopped.
8.93 nats (approx. ln 4096) at initialization -> approx. 3.33 nats at the final step, over 11,933 steps
See it in the paper
“Training loss fell from 8.93 (≈ln 4096 at initialization) to ≈3.33 at the final step and was still decreasing when the step budget was exhausted.”Appendix / Training details / Transformer
The model's total learned improvement over guessing uniformly is about 4.94 nats per token.
uniform-guess cost ~8.32 nats; test-set loss 3.38 nats; difference ~4.94 nats
See it in the paper
“predicting uniformly over the 4,096-token vocabulary costs … 8.32 nats per token, and the trained model's loss on the test set (the 10th shard of data) is 3.38 , so the weights collectively deliver about 4.94 nats per token.”Appendix / ROPE plots
Nearly all virtual weights in the full model have some nonzero measured effectiveness.
331,350,016 total weights; 315,487,613 with nonzero effectiveness
See it in the paper
“331,350,016 weights, 315,487,613 with nonzero effectiveness”Appendix / Fisher mass distribution
Total positive (helpful) effectiveness mass outweighs total negative (harmful) effectiveness mass by more than 2x.
sum of positive-weight effectiveness = 3.62; sum of negative-weight effectiveness = 1.59
See it in the paper
“The sum of effectiveness across all positive weights is 3.62. The analogous sum across negative weights is 1.59.”Appendix / Fisher mass distribution
In the sampled weights, total positive helpfulness mass far exceeds total negative helpfulness mass.
sum of positive helpfulness = 1.44x10^-4; sum of negative helpfulness magnitudes = 1.12x10^-5
See it in the paper
“The sum of positive helpfulness values in this sample is 1.44×10-4. The sum of negative helpfulness magnitudes is 1.12×10-5.”Appendix / Helpfulness mass distribution
The largest raw virtual weight from the token 'IN' is both far less effective than other weights from the same token and actively harmful.
~3 orders of magnitude below the most effective weights from 'IN'; the predicted completion never occurs in the training set
See it in the paper
“That weight's Fisher effectiveness, however, is around three orders of magnitude below that of the most effective weights from the same token. It is also harmful; across the model's entire training set, " utions " never once follows " IN ".”Effectiveness surfaces helpful weights but does not isolate them
Effectiveness varies far more sharply across weights than raw virtual-weight magnitude does.
median virtual weight ~1/3 of the maximum magnitude; median effectiveness ~10,000x less than the maximum effectiveness
See it in the paper
“The median magnitude of virtual weight is approximately one third of the magnitude of the largest weight, while the median effectiveness of a virtual weight is 10,000× less than the maximum effectiveness.”Effectiveness concentrates helpful and harmful weights (IN Tokens → Logits)
In a newline-detecting feature, its two position-based inputs almost exactly cancel each other out.
OLS slope = -0.993, r^2 = 0.912; combined contribution ~30% of the feature's effective activation threshold
See it in the paper
“the input from the Tokens→Features and OV paths are strongly anti-correlated (OLS slope= -0.993, r^2 =0.912), and the largest contribution from their sum only constitutes ~30% of the feature's effective threshold (including the JumpReLU threshold and the bias parameter).”Effectiveness surfaces helpful weights but does not isolate them
The transformer was trained for a single pass over tens of millions of unique tokens with no repeated data.
~9.8x10^7 unique tokens (95,464 unique sequences), 11,933 steps, single pass
See it in the paper
“The model trained on ≈9.8×10⁷ unique tokens (95,464 unique sequences) over its 11,933 steps in a single pass with no data repetition.”Appendix / Training details / Transformer
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
Large raw virtual-weight magnitude is not a reliable signal of which weights matter to the model's behavior.
“First, helpful and harmful weights are scattered across the entire range of virtual weight magnitudes.”
“Virtual weight shows a weak relationship to helpfulness (panels 1 and 2); for this set, the virtual weight highlights few of the most important connections.”
Effectiveness concentrates helpful and harmful weights (IN Tokens → Logits)Ineffective virtual weights are abundant and can be pruned cheaply with little cost to model performance.
“Second, ineffective virtual weights are abundant and can be easily removed: pruning the least effective 70% only worsens the model's loss by 0.01 nats, and pruning 85% costs 0.1.”
“This costs approximately 0.01 nats at 70% sparsity (30% density) and under 0.1 nats at 85% sparsity (15% density).”
The model is still dense in this basisThe model concentrates its most effective weights in helpful directions and confines harmful weights to a lower-effectiveness range.
“we find that the model puts its most effective weights in helpful directions and confines its harmful ones to a lower-effectiveness range.”
“Only helpful weights remain in the highest effectiveness regime. The most effective helpful weight out-measures any harmful weight by an order of magnitude or more within each weight family.”
The most effective weights are helpfulEven after identifying interference weights, the number of remaining helpful weights is still too large for this basis to yield a sparse, fully interpretable model.
“our virtual weight model contains more helpful weights than parameters in the original transformer, even when constrained to the most effective and helpful.”
“Across the sample, roughly half (47.6%) of all weights have positive mean helpfulness (with 12.7% dead)... Since our VW model increases the overall number of parameters by two orders of magnitude, we're still left with tens of millions of weights to interpret for a single-layer model.”
The model is still dense in this basisThis is the first demonstration of an interference weight inside an actual trained transformer, verified by its measured effect on training loss (rather than only in a toy model).
“this note is the first place, to our knowledge, that an interference weight like this one has been demonstrated inside a trained transformer by measuring their effect on the training loss .”
“This token never follows " IN " in the training set, so every time this virtual weight affects the output, it only makes the model's loss worse.”
IntroductionSorting or filtering weights by Fisher effectiveness does not fully separate helpful from harmful weights.
“Fisher effectiveness reliably highlights the most helpful weights, but it does not fully remove interference weights.”
“The most harmful weights remain intermingled with the helpful ones across the high-effectiveness range, even though most of the weights on the large-effectiveness end are helpful.”
Harmful weights overlap with helpful ones (Feature 3013 → Logits)How they frame it, and what they want next
Their framing
The authors frame this as the first concrete, trained-model evidence that weight superposition really does produce harmful or irrelevant 'interference weights,' rather than only a phenomenon shown in toy models. They present their two-number scoring scheme (effectiveness and helpfulness) as a useful but incomplete tool for separating real circuitry from noise, and they frame the leftover density after pruning as evidence that the real bottleneck for future work is finding a better basis for the decomposition, not a better pruning threshold.
Register: The prose is measured and self-qualifying: the authors repeatedly hedge with phrases like 'we suspect,' 'may,' and 'we think this says less about... than about,' and they plainly state what they were 'unable' to achieve rather than reframing the negative pruning result as success.
Where they hedge
“This work focuses on a small transformer to leverage the most accurate and expensive tools we have to identify interference weights.”Discussion
“Fisher effectiveness and helpfulness may be useful for interpreting logit effects in other settings, but they do not scale nicely to frontier models across other families, so further development of scalable proxies like ERA and TWERA may be useful for interpreting model components.”Discussion
“We were unable to reach a highly sparse and interpretable model using Fisher effectiveness as the filtering criterion, and we suspect, based on our helpfulness computations, that no saliency scheme will perform much better.”Discussion
“identifying them turns out to be necessary but not sufficient for reading global circuits.”Discussion
“The transcoder we trained also has room for improvement as it contains several polysemantic features, and the four attention heads in this model are also likely to be polysemantic”Discussion
What they say it means
- Progress toward a fully sparse, interpretable model likely requires a better feature/parameter basis, not a better pruning threshold.
the paper’s words
“We suggest that finding a new basis for the VW model's weights is the most promising route to further progress.”The model is still dense in this basis
- Studying interference in frontier-scale models will likely require cheaper proxy metrics rather than the full effectiveness/helpfulness computation used here.
the paper’s words
“they do not scale nicely to frontier models across other families, so further development of scalable proxies like ERA and TWERA may be useful for interpreting model components.”Discussion
- Identifying and removing interference weights is a necessary but not sufficient step toward reading circuits directly off a model's global weights.
the paper’s words
“identifying them turns out to be necessary but not sufficient for reading global circuits.”Discussion
What they call for next
- Future work should search for a different basis (e.g., organized by language or part of speech) in which the model's computation is actually sparse.
the paper’s words
“future work may uncover better bases for this task.”Discussion
- Future work should develop scalable proxy metrics for effectiveness and helpfulness that could apply to frontier-scale models.
the paper’s words
“further development of scalable proxies like ERA and TWERA may be useful for interpreting model components.”Discussion
Limitations they state
“This work focuses on a small transformer to leverage the most accurate and expensive tools we have to identify interference weights.”Discussion
“In the derivations below we take these uses as "independent" and our expressions are not exact when the same token appears multiple times.”Appendix / QK weights
“For tractability the indirect route through the transcoder treats features as independent, using the full variance for each feature but dropping cross-feature covariances. The covariance between the direct and indirect routes is also dropped.”Appendix / QK weights
“The transcoder we trained also has room for improvement as it contains several polysemantic features, and the four attention heads in this model are also likely to be polysemantic”Discussion
“It is difficult to say which level of helpfulness mass is a better guide, particularly when we also expect nonlinear effects from removing multiple weights together.”The model is still dense in this basis
Moves worth stealing
Opens with a single, concretely-worked motivating example (a word-completion prediction) before stating any general claims, so the reader feels the underlying puzzle before the metrics are introduced.
“We can decompose the prediction to complete ACETYLCHOLINE into the separate paths that feed the output and read off what each contributes.”
States the paper's main findings as a short numbered list immediately after the introduction's setup, giving the punchline before the detailed derivations.
“Expanding upon this result, we then present three major findings.”
Explicitly declines to over-claim novelty for an incremental metric change, distinguishing 'this is an improvement' from 'this is worth treating as a discovery.'
“We don't encourage readers to read much into our change from ERA or TWERA to Fisher effectiveness.”
Uses a dedicated 'Related work' section that names specific prior methods and states precisely how the new metric descends from or differs from each, rather than a generic citation dump.
“our Fisher effectiveness is a direct descendant of these criteria, applied to virtual weights and accumulated over the data distribution.”
Reframes a negative result (failing to reach a sparse model) not as a dead end but as evidence about the choice of basis, stating the failure plainly and then immediately offering a specific alternative explanation.
“We think this says less about the potential for a sparse interpretation of this model than about the basis in which we study it.”
Where else this leads
Same people
- Verbalizable Representations Form a Global Workspace in Language Models Anthropic
shares Joshua Batson - HeadVis Anthropic
shares Joshua Batson - Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations Anthropic
shares Joshua Batson
Same territory
- Verbalizable Representations Form a Global Workspace in Language Models Anthropic
mechanistic interpretability - HeadVis Anthropic
mechanistic interpretability
Published alongside it
The nearest publications in time, across all three labs.
- Automated Researchers Can Mitigate Well-Characterized Alignment Failures Anthropic
2026-08-15 - Fine-Tuned Lie Detectors Failed to Generalize Anthropic
2026-08-15 - Introducing the Conceptual Reasoning Index Anthropic
2026-08-15 - TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic
2026-08-15
What this page was built from
Working from a full plain-text extraction of the published Transformer Circuits web page, including its Appendix; interactive figures and plots are represented only by their captions and surrounding prose, not their underlying visual/numeric data.