Modular Pretraining Enables Access Control
GRAM trains one model with switchable auxiliary modules that approximate multiple separately data-filtered models, isolating dangerous knowledge like virology or cybersecurity from general capabilities.
It shows a concrete, tested approach to capability-level access control that could let one trained model serve trusted and untrusted users differently, without training separate models.
Ethan Roland · Murat Cubuktepe · Erick Martinez · Stijn Servaes · Keenan Pepper · Mike Vaiana · Diogo Schwerz de Lucena · Judd Rosenblatt · Addie Foote · Cem Anil · Alex Cloud — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
“Frontier AI models have knowledge that could be misused for nefarious purposes. To address this risk, we introduce Gradient Routed Auxiliary Modules (GRAM), a method for isolating dangerous knowledge to specific modules within a language model. These modules can be switched on or off to control what the model knows, making it possible to restrict or extend access to the most sensitive model capabilities based on user need and trust. In our experiments, we find evidence that a single model trained in this way can approximate multiple models, each trained with a different category of dangerous data filtered out, and this ability holds for models ranging from 50M to 5B parameters. This research is preliminary and has not been applied to production models at Anthropic.”
Researchers built a training method called GRAM that adds small, switchable modules to a language model so specific pieces of risky knowledge — like virology or cybersecurity details — can be turned on or off after the model is trained. Instead of training a separate model for every combination of who should get which sensitive capability, one model can be reconfigured to match many differently-filtered models. Across model sizes from 50 million to 5 billion parameters, this works about as well as training filtered models from scratch, using only a fraction of the compute.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
Gradient Routed Auxiliary Modules (GRAM)
“GRAM augments the MLP layer in each block of a Transformer by introducing small auxiliary modules, which are just additional neurons. During training, the modules that participate in forward and backward passes are chosen based on the data in each batch.”The Method
In plain terms: A training method that adds small extra pieces to a model, one per sensitive topic, that only learn from and predict that topic's data, so they can later be deleted to remove just that knowledge.
Access control (capability-level)
“An alternative approach is access control at the level of individual capabilities. For example, a deployment that includes advanced virology knowledge for a vetted biosecurity lab and excludes it everywhere else, with general performance unchanged in both cases.”Introduction
In plain terms: Letting different users get different specific capabilities from the same underlying model, rather than an all-or-nothing choice between model versions.
Compute Ratio
“we calculate model performance in terms of Compute Ratio , a compute-adjusted version of loss. It measures, for a given model and data domain, how long it took the all-data baseline model to reach the same loss while training, relative to the length of a whole training run. A value of 1.0 matches the baseline and a value of 0.5 indicates the model performs as well as the baseline after finishing 50% of training.”The Method
In plain terms: A score comparing how a model does on some data versus a baseline model trained on everything, expressed as how far into the baseline's training run it takes to match that performance.
Adversarial elicitation
“We then test whether the removal survives “adversarial elicitation” from an adversary fine-tuning on malicious data.”Access Control on Real Dual Use Data
In plain terms: Testing whether an attacker can bring back a supposedly removed capability just by fine-tuning the model on related data.
Entangled capabilities
“Some general capabilities (like knowledge of biology), might be so closely related to dual use capabilities (like virology), that there is no way to cleanly separate them.”Discussion
In plain terms: A risk that dangerous knowledge and ordinary useful knowledge are so intertwined that removing one damages the other.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Built the GRAM architecture by adding small auxiliary modules to the MLP layer of every Transformer block, one module per specialized (e.g., dual-use) dataset.
Trace this step to the paper
“GRAM augments the MLP layer in each block of a Transformer by introducing small auxiliary modules, which are just additional neurons.”The Method
- Routed each training batch to the module matching its data category, activating the biology module for biology text and so on.
Trace this step to the paper
“During training, the modules that participate in forward and backward passes are chosen based on the data in each batch.”The Method
- On general-purpose 'core' data, occasionally activated a random auxiliary module so the model performs well regardless of which modules are enabled at inference.
Trace this step to the paper
“When training on general-purpose “core” data (right), we occasionally enable a random auxiliary module to ensure good performance on general data regardless of the inference-time configuration of the model.”The Method
- Controlled how strongly a capability was isolated into its module by adjusting how often the general-purpose weights were frozen during that module's updates.
Trace this step to the paper
“By adjusting how often those general-purpose weights are frozen, we can control how well biology is isolated.”The Method
- Ran a synthetic-data experiment: pretrained a 26M-parameter, 8-layer Transformer on a children's-story dataset with 4 of 48 topics assigned auxiliary modules, compared against 5 separately trained data-filtered models.
Trace this step to the paper
“a synthetic corpus of 2M short children's stories, each labeled with one of 48 topics … We choose four topics to serve as auxiliary capabilities and pool the remaining 44 topics into the core dataset, then pretrain a small Transformer (26M parameters, 8 layers) with one auxiliary module per auxiliary topic.”GRAM Matches Data Filtering
- Ran a realistic-data experiment: trained 800M-parameter models on general text/code/papers plus four dual-use domains, comparing GRAM against data filtering, LoRA fine-tuning, and the MaxEnt post-hoc unlearning baseline.
Trace this step to the paper
“We train 800M-parameter models on web text, code, and scientific papers, plus four dual use domains: virology, cybersecurity, nuclear physics, and specialist code.”Access Control on Real Dual Use Data
- Tested whether removed capabilities could be restored by an adversary fine-tuning the model on malicious data.
Trace this step to the paper
“We then test whether the removal survives “adversarial elicitation” from an adversary fine-tuning on malicious data.”Access Control on Real Dual Use Data
- Ran a scaling study: trained seven GRAM models from 50M to 5B parameters at Chinchilla-optimal dataset sizes, holding dual-use data at a constant 1% share, plus one LoRA and one data-filtered comparison model.
Trace this step to the paper
“we train seven GRAM models at seven sizes from 50M to 5B parameters, growing the datasets according to Chinchilla-optimal sizing, and holding dual use data at 1% of the general dataset size throughout.”Modularization Works Across Scales
- Tested composability by enabling all four auxiliary modules simultaneously at inference and comparing to summing multiple LoRA adapters at once.
Trace this step to the paper
“In this experiment, we see what happens if we enable multiple auxiliary modules at inference time.”Composability
- Tested robustness to partial labeling by removing labels from half the training data; GRAM trained on unlabeled batches with all modules active, while filtering/LoRA treated unlabeled data as general-purpose.
Trace this step to the paper
“To test the performance of GRAM in more realistic settings, we removed the labels from half of all data. For filtering and LoRA, we treat unlabeled data as if it was general-purpose data. For GRAM, we train on unlabeled data while keeping all modules active.”Isolation Under Partial Labeling
Exactly what was run, and how
| Model | Developer | Temp | Effort / reasoning | Deployment | Other settings |
|---|---|---|---|---|---|
| 26M-parameter Transformer (8 layers) | This paper's authors (AE Studio, with Anthropic) | not reported | not reported | local weights | Pretrained from scratch on the SimpleStories synthetic dataset (2M children's stories, 48 topics), one auxiliary module per auxiliary topic. |
| 800M-parameter language model | This paper's authors (AE Studio, with Anthropic) | not reported | not reported | local weights | Trained on web text, code, and scientific papers plus four dual-use domains (virology, cybersecurity, nuclear physics, specialized code); dual-use data was about 40M tokens per domain against 16B general tokens (~0.25% each). |
| GRAM models across scales (50M–5B parameters, 7 sizes) | This paper's authors (AE Studio, with Anthropic) | not reported | not reported | local weights | Chinchilla-optimal dataset sizing per model size; dual-use data held at 1% of the general dataset size throughout; a LoRA model and a single data-filtered model were also trained for comparison. |
Source for 26M-parameter Transformer (8 layers) settings
“pretrain a small Transformer (26M parameters, 8 layers) with one auxiliary module per auxiliary topic.”GRAM Matches Data Filtering
Source for 800M-parameter language model settings
“We train 800M-parameter models on web text, code, and scientific papers, plus four dual use domains: virology, cybersecurity, nuclear physics, and specialist code.”Access Control on Real Dual Use Data
Source for GRAM models across scales (50M–5B parameters, 7 sizes) settings
“we train seven GRAM models at seven sizes from 50M to 5B parameters, growing the datasets according to Chinchilla-optimal sizing, and holding dual use data at 1% of the general dataset size throughout.”Modularization Works Across Scales
What they reported — and what they left out
The paper reports parameter counts, layer count for the smallest model, training-corpus composition, dual-use data fractions, and Chinchilla-optimal scaling for its own custom-trained Transformers, but never states learning rate, optimizer, batch size, tokenizer, temperature, or any inference-time sampling settings beyond which auxiliary modules were active.
The numbers they report
GRAM can be reconfigured to match the performance of each of five separately trained, data-filtered models from a single training run.
See it in the paper
“indicating our method is able to approximate the performance of five distinct filtered models trained on different data, despite only requiring a single training run.”GRAM Matches Data Filtering
The synthetic-story experiment isolated a small number of topics as auxiliary capabilities within a much larger topic set.
48 total topics; 4 auxiliary topics; 44 core topics; 26M-parameter, 8-layer Transformer
See it in the paper
“a synthetic corpus of 2M short children's stories, each labeled with one of 48 topics … We choose four topics to serve as auxiliary capabilities and pool the remaining 44 topics into the core dataset”GRAM Matches Data Filtering
In the realistic dual-use experiment, the sensitive data made up only a small share of total training tokens.
16B general tokens vs. roughly 40M tokens per risky domain (about 0.25% each)
See it in the paper
“The dual use data is a small fraction of training: 16B general tokens against roughly 40M per risky domain, about 0.25% each.”Access Control on Real Dual Use Data
Ablating GRAM's modules or deleting LoRA's adapters removes capabilities nearly as well as never having trained on that data, while both retain better performance than filtering on kept domains.
See it in the paper
“ablating GRAM's modules or deleting LoRA’s adapters removes capabilities nearly as effectively as never training on the data at all. On retained domains, GRAM and LoRA achieve better performance than filtering.”Access Control on Real Dual Use Data
Filtering, LoRA, and GRAM all resist an adversary's attempt to recover removed capabilities via malicious fine-tuning, but the MaxEnt unlearning baseline does not.
See it in the paper
“Filtering, LoRA, and GRAM all show strong robustness to malicious fine-tuning. … In contrast, MaxEnt, an unlearning method used to modify pre-existing models, recovers to near the performance of the all-data baseline”Access Control on Real Dual Use Data
A single GRAM training run yields as many deployable configurations as filtering would need separate runs for.
1 GRAM run produces 5 configurations that filtering needs 5 separate runs to produce
See it in the paper
“Filtering needs five training runs to produce these five configurations. GRAM produces all five from one run and switches between them by deleting weights at inference.”Access Control on Real Dual Use Data
The scaling study covered seven model sizes with a constant dual-use data share.
7 model sizes, 50M–5B parameters, dual-use data held at 1% of the general dataset
See it in the paper
“we train seven GRAM models at seven sizes from 50M to 5B parameters … holding dual use data at 1% of the general dataset size throughout.”Modularization Works Across Scales
Both capability removal and robustness to adversarial elicitation improve as models are scaled up.
See it in the paper
“For all methods, both capability removal and robustness to elicitation improve with scale.”Modularization Works Across Scales
GRAM and LoRA match data-filtering performance at every tested scale while needing far less training compute than filtering.
GRAM and LoRA require about one-fifth of filtering's training compute
See it in the paper
“GRAM and LoRA track filtering at every scale and both require only a fifth of filtering's training compute.”Modularization Works Across Scales
GRAM's capabilities compose cleanly when multiple modules are enabled together, unlike LoRA adapters, which degrade when combined; this multiplies how many configurations one run can serve.
4 GRAM modules yield 16 possible configurations, versus 16 separate runs needed for filtering
See it in the paper
“with all four modules on, virology performance matches the run where virology alone was retained. In contrast, summing multiple LoRA adapters degrades performance in every category. … four GRAM modules give sixteen possible configurations, where filtering would need sixteen different runs.”Composability
When half the training data is unlabeled, GRAM isolates dual-use capabilities more effectively than data filtering or LoRA.
50% of training data left unlabeled
See it in the paper
“With only 50% of training data labeled, GRAM reaches a much lower forget compute ratio than either data filtering or LoRA, indicating stronger isolation of dual use capabilities.”Isolation Under Partial Labeling
The Compute Ratio metric is anchored so that 1.0 equals full baseline performance and 0.5 equals reaching that performance at the training run's halfway point.
1.0 = matches baseline; 0.5 = baseline-level performance at 50% of training
See it in the paper
“A value of 1.0 matches the baseline and a value of 0.5 indicates the model performs as well as the baseline after finishing 50% of training.”The Method
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
GRAM approximates the performance of multiple separately data-filtered models at the cost of a single training run.
“we develop Gradient-Routed Auxiliary Modules (GRAM) , a method that approximates the performance of multiple data filtered models at the cost of a single training run.”
“By reconfiguring one model, GRAM can approximate the performance of each of the five distinct data filtered runs.”
GRAM Matches Data FilteringGRAM composes multiple retained capabilities more cleanly than LoRA.
“GRAM composes capabilities better than LoRA.”
“with all four modules on, virology performance matches the run where virology alone was retained. In contrast, summing multiple LoRA adapters degrades performance in every category.”
ComposabilityGRAM isolates dual-use capabilities better than data filtering and LoRA when training labels are only partially available.
“GRAM isolates capabilities better than the alternatives under partial labeling.”
“GRAM has a much larger drop in performance after deleting auxiliary parameters, indicating better isolation of auxiliary capabilities into the relevant module.”
Isolation Under Partial LabelingMaxEnt-style post-hoc unlearning suppresses dangerous capabilities rather than truly removing them, unlike GRAM, LoRA, and filtering.
“MaxEnt does not truly remove capabilities but instead learns to suppress them.”
“Filtering, LoRA, and GRAM all show strong robustness to malicious fine-tuning. In contrast, MaxEnt, an unlearning method used to modify pre-existing models, recovers to near the performance of the all-data baseline”
Access Control on Real Dual Use DataCapability removal via GRAM, LoRA, and filtering all improve with model scale.
“Capability removal improves with scale.”
“The gap on the forgotten domain widens with scale: bigger models fall further behind the all-data baseline on virology when the capability is removed, and recover less performance when fine-tuned on virology data.”
Modularization Works Across ScalesHow they frame it, and what they want next
Their framing
The authors frame GRAM as an early but promising step toward capability-level access control, explicitly labeling the work preliminary and not yet used on production Anthropic models. Rather than closing with a confident conclusion, they structure the entire Discussion as a list of self-posed open questions about entanglement, production feasibility, downstream validity, generalization, and the real difference between GRAM and LoRA.
Register: The authors state their own experimental comparisons in confident, direct terms ('we find', 'GRAM matches', 'GRAM composes capabilities better than LoRA'), but wrap nearly every headline result in an explicit caveat about scale, production-readiness, or the narrowness of the specific experiment.
Where they hedge
“This research is preliminary and has not been applied to production models at Anthropic.”tl;dr
“Our scaling experiments show clear trends up to 5B parameters, but we didn’t verify whether these trends continue at frontier scales or hold in a production setting.”Discussion
“Implementing GRAM in a production setting could be difficult and add a lot of complexity.”Discussion
“We didn't test meaningfully different settings, so we can't tell whether the methods are similar in general or only in this instance.”Discussion
“these are two isolated experiments, so we can’t be sure these results will hold in general.”Discussion
“In absolute terms, the bigger models are still better at the forgotten capabilities.”tl;dr
What they say it means
- If it works at scale, capability-level access control could let one trained model serve different users different sensitive capabilities based on trust, instead of forcing an all-or-nothing choice between full and weakened models.
the paper’s words
“These modules can be switched on or off to control what the model knows, making it possible to restrict or extend access to the most sensitive model capabilities based on user need and trust.”tl;dr
- If dangerous and general knowledge turn out to be too entangled to separate, then data-filtering-based access control approaches (including GRAM) may not be a viable defense at all.
the paper’s words
“In practice, this would mean that data filtering itself is ineffective, because the tradeoff between retain and forget capabilities is too steep.”Discussion
What they call for next
- Scale modular pretraining methods like GRAM up to larger, more realistic training settings.
the paper’s words
“Scaling modular pretraining to larger and more realistic settings;”Discussion
- Work out how to instruction-tune a model that was pretrained with GRAM's modular structure.
the paper’s words
“Figuring out how to instruction-tune a model trained with GRAM;”Discussion
- Study how modular pretraining behaves when data labels are imperfect, including when labeling errors are not random or independent of the data.
the paper’s words
“Better understanding how modular pretraining works when data is imperfectly labeled, including when the label errors aren’t random and independent from the data.”Discussion
Limitations they state
“This research is preliminary and has not been applied to production models at Anthropic.”tl;dr
“Our scaling experiments show clear trends up to 5B parameters, but we didn’t verify whether these trends continue at frontier scales or hold in a production setting.”Discussion
“Implementing GRAM in a production setting could be difficult and add a lot of complexity.”Discussion
“Our experiments only use loss-based evals, which are generally predictive of performance on downstream tasks, but may not capture everything that matters.”Discussion
“We didn't test meaningfully different settings, so we can't tell whether the methods are similar in general or only in this instance.”Discussion
“these are two isolated experiments, so we can’t be sure these results will hold in general.”Discussion
Moves worth stealing
Leads with a concise 'tl;dr' that gives the risk motivation, the method name, and the headline result before any technical detail, functioning as an accessible substitute for a formal abstract.
“Frontier AI models have knowledge that could be misused for nefarious purposes. To address this risk, we introduce Gradient Routed Auxiliary Modules (GRAM)”
States the work's preliminary, not-yet-production status directly in the summary rather than deferring the caveat to a limitations section, tempering excitement about a safety-relevant capability up front.
“This research is preliminary and has not been applied to production models at Anthropic.”
Structures the entire Discussion as a numbered set of open questions the authors pose to themselves, honestly naming what's unresolved instead of closing on a triumphant summary.
“Do entangled capabilities make data filtering itself infeasible?”
Names a concurrently developed competing method by name and points readers to it rather than ignoring adjacent work, situating the post within an active research conversation.
“Readers may also be interested in NULLs , a similar method developed concurrently.”
Defines its core metric with a worked numeric example baked into the definition itself, so readers can interpret every later chart without further explanation.
“A value of 1.0 matches the baseline and a value of 0.5 indicates the model performs as well as the baseline after finishing 50% of training.”
Where else this leads
Same territory
- Project Pilot: Can AI control a drone? Anthropic
dual-use risk
Published alongside it
The nearest publications in time, across all three labs.
- Agentic Misalignment in Summer 2026 Anthropic
2026-07-15 - Verbalizable Representations Form a Global Workspace in Language Models Anthropic
2026-07-15 - GPT-Red: Unlocking Self-Improvement for Robustness OpenAI
2026-07-15 - Quantifying the Salience of Geo-Cultural Values for Pluralistic Safety Alignment Google DeepMind
2026-07-10
What this page was built from
Extracted from the lab's blog-post writeup (Alignment Science Blog), which uses a 'tl;dr' in place of a formal Abstract heading (reproduced verbatim in the abstract field); the byline in the text itself gives the date as July 8, 2026, which is used here as pub_date, differing slightly from the corpus manifest's 2026-07-15 (listed only at month precision); numeric results are conveyed mostly through bar-chart figures whose captions and surrounding prose are quoted here, but the charts' exact underlying values are not reproduced as text in the source file.