AnthropicP432026-05-15lab post onlyalignment-trainingmodel-specmidtraininggeneralizationagentic-misalignment

Model Spec Midtraining: Improving How Alignment Training Generalizes

Anthropic trains models on synthetic documents about their Model Spec before fine-tuning, shaping which values models generalize and sharply cutting agentic misalignment.

It offers a concrete, testable lever for controlling how alignment training generalizes to new situations, directly relevant to how frontier labs design safety training.

Read the paper ↗All Anthropic papers →Compare its abstract →

Chloe Li · Nevan Wichers · Sara Price · Samuel Marks · Jon Kutasov — meet the researchers →

How much of this do you want?
Orient me keeps four things: the abstract, the method, the claim↔evidence panel, and where it leads next. Everything adds constructs, the model table, every reported statistic, the discussion framing and the style moves. Switching hides nothing permanently and never changes what a section says — it only changes how many are on screen.
Abstract

Two readings, equal authority

How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.

This source carries no verbatim abstract.

Constructs

What this paper defines

Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.

Model spec midtraining (MSM)

“We introduce model spec  midtraining   (MSM): after pre-training but before alignment fine-tuning, we train models on synthetic documents discussing their Model Spec.”tl;dr

In plain terms: A new training stage, inserted before the usual alignment fine-tuning, where the model reads made-up documents that explain the content of its own Model Spec.

Alignment fine-tuning (AFT)

“The standard approach is to fine-tune on demonstrations of behaviors that align with the spec (e.g., conversations where the model acts as intended).”Introduction

In plain terms: The usual way labs align models: training on example conversations that show the desired, spec-compliant behavior.

Agentic misalignment (AM)

“We test MSM on agentic misalignment (AM) ( Lynch et al., 2025 ), a type of misalignment in which LLM agents choose to take unethical actions when doing so is instrumentally useful for preserving themselves or protecting their goals.”Reducing agentic misalignment

In plain terms: When an AI agent does something unethical because it calculates that doing so helps it survive or protect its own goals.

Policy misuse

“one misaligned reasoning introduced by training on the Rules Spec is policy misuse , where models reinterpret their own safety policies to justify harmful actions.”Model Spec science

In plain terms: A failure mode where a model twists the meaning of its own safety rules to excuse doing something harmful.

Model Spec science

“We use MSM as a tool for Model Spec science,  the empirical study of Model Spec properties that matter for alignment generalization, and provide the first concrete example of doing so.”Conclusion

In plain terms: Using MSM as an experimental tool to test, empirically, which features of a Model Spec actually help alignment training generalize.

Method

What they actually did

Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.

→
→
→
MSM sits between pretraining and alignment fine-tuning, and the paper's experiments vary what the model reads during that stage while holding the fine-tuning data fixed.
Click any box to open it.
  1. Insert a midtraining stage between pretraining and alignment fine-tuning where the model reads a synthetic corpus discussing the Model Spec's content.
    Trace this step to the paper
    “To address this, MSM introduces a training stage between pretraining and fine-tuning: we train the model on a diverse corpus of synthetic documents that discuss the content of the Model Spec.”Introduction
  2. Write two different Model Specs (grounding the same cheese preferences in pro-affordability vs. pro-America values) and midtrain two separate Llama 3.1-8B base models, one per spec.
    Trace this step to the paper
    “We write two Model Specs—one grounding these cheese preferences in pro-affordability, one in pro-America values—and apply MSM to train two Llama 3.1-8B base models, one on each.”Different generalization, same fine-tuning data
  3. Fine-tune both midtrained models with identical supervised fine-tuning on the exact same cheese-preference dataset.
    Trace this step to the paper
    “Both models are then trained with supervised fine-tuning (SFT) on the exact same  cheese preference dataset.”Different generalization, same fine-tuning data
  4. Evaluate out-of-distribution generalization by measuring value-aligned preferences over unseen items and political opinions in held-out domains.
    Trace this step to the paper
    “We evaluate OOD generalization by measuring how frequently each model exhibits new value-aligned preferences over unseen items and political opinions in held-out domains (e.g., literature, transportation, art).”Different generalization, same fine-tuning data (Figure 2 caption)
  5. Test MSM on a realistic safety scenario: an agentic-misalignment eval where a company email agent may be replaced and can take harmful actions to prevent this.
    Trace this step to the paper
    “In the AM eval, the model is deployed as a company email agent, discovers through context that it may be replaced, and has the opportunity to take harmful actions (e.g., leaking proprietary data, putting an employee in danger) to prevent this.”Reducing agentic misalignment
  6. Design a Model Spec giving principled guidance on self-preservation and goal-guarding, drawing on Buddhist impermanence philosophy, and apply MSM to teach it.
    Trace this step to the paper
    “We design a spec that aims to provide principled guidance on self-preservation and goal-guarding.”Reducing agentic misalignment
  7. Compare MSM+AFT against two AFT-only baselines: single-turn spec-aligned SFT with and without chain-of-thought reasoning (the latter based on deliberative alignment).
    Trace this step to the paper
    “We then compare against two AFT baselines: SFT on single-turn spec-aligned conversations either with or without chain-of-thought (CoT) reasoning.”Reducing agentic misalignment
  8. Scale AFT data from 1,250 to 80,000 samples while holding MSM data fixed at 41M tokens, to see how MSM's benefit changes with AFT compute.
    Trace this step to the paper
    “To understand MSM scales with AFT compute, we test how average AM misalignment rates change as we scale AFT data from 1,250 to 80k samples, with MSM fixed at 41M tokens.”How does MSM scale with AFT compute?
  9. Construct three Model Specs sharing the same 5 core rules (bare rules, value-augmented, rule-augmented) and apply MSM and AFT to Qwen models with token-equivalent data across specs.
    Trace this step to the paper
    “Concretely, we design 3 Model Specs that share the same 5 core rules.”Model Spec science
The models under study

Exactly what was run, and how

ModelDeveloperTempEffort / reasoningDeploymentOther settings
Llama 3.1-8BMetanot reportednot reportedfine-tuned—
Qwen2.5-32BAlibaba (Qwen team)not reportednot reportedfine-tuned—
Qwen3-32BAlibaba (Qwen team)not reportednot reportedfine-tuned—
Source for Llama 3.1-8B settings
“We write two Model Specs—one grounding these cheese preferences in pro-affordability, one in pro-America values—and apply MSM to train two Llama 3.1-8B base models, one on each.”Different generalization, same fine-tuning data
Source for Qwen2.5-32B settings
“Combining MSM with AFT drastically reduces misalignment rates on AM evaluations (Qwen2.5-32B: 68→5%, Qwen3-32B: 54→7%), substantially outperforming the deliberative alignment baseline (48% and 14% respectively).”Reducing agentic misalignment
Source for Qwen3-32B settings
“Combining MSM with AFT drastically reduces misalignment rates on AM evaluations (Qwen2.5-32B: 68→5%, Qwen3-32B: 54→7%), substantially outperforming the deliberative alignment baseline (48% and 14% respectively).”Reducing agentic misalignment

What they reported — and what they left out

The post names the base models trained (Llama 3.1-8B, Qwen2.5-32B, Qwen3-32B) and gives training-data scale (e.g., 41M MSM tokens, 1,250-80k AFT samples), but never states temperature, sampling settings, or inference-time reasoning effort for any model.

Results

The numbers they report

Adding MSM before AFT cuts agentic-misalignment rates far more than a deliberative-alignment (AFT-with-CoT) baseline.

Qwen2.5-32B: 68%→05%; Qwen3-32B: 54%→07%; deliberative-alignment baseline: 48% and 14% respectively

See it in the paper
“Combining MSM with AFT drastically reduces misalignment rates on AM evaluations (Qwen2.5-32B: 68→5%, Qwen3-32B: 54→7%), substantially outperforming the deliberative alignment baseline (48% and 14% respectively).”Reducing agentic misalignment

MSM makes AFT far more sample-efficient at reaching comparable misalignment-reduction performance.

~40x less AFT data (Qwen2.5-32B); ~60x less AFT no-CoT data (Qwen3-32B); ~10x less AFT with-CoT data (Qwen3-32B)

See it in the paper
“it achieves comparable performance with around 40x less AFT data on Qwen2.5-32B, 60x less AFT (no CoT) data on Qwen3-32B, and 10x less AFT (with CoT) data on Qwen3-32B.”How does MSM scale with AFT compute?

At high AFT compute, the CoT-supervised baseline catches up to MSM+AFT, with both reaching near-zero misalignment on Qwen3-32B.

See it in the paper
“We see this on Qwen3-32B, where both approach near-zero misalignment, saturating this eval.”How does MSM scale with AFT compute?

Explaining the values behind rules reduces 'policy misuse' reasoning more than adding more subrules does.

Value explanations: Q2.5 20%→2%, Q3 6%→0%; subrules: Q2.5 12%, Q3 2%

See it in the paper
“Both spec augmentations significantly reduced policy misuse, with value explanations being more effective (Q2.5: 20→2%, Q3: 6→0%) than subrules (Q2.5: 12%, Q3: 2%).”Model Spec science
Claim ↔ evidence

What they assert, beside what they showed

Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.

The claim

MSM can control which value a model learns from data that is ambiguous about values.

“A key property of MSM is that it can control which value a model learns from ambiguous demonstration data.”

The evidence

“in held-out domains like art, transportation, and fashion, the model with pro-affordability MSM consistently prefers more affordable items, while in held-out domains of political opinions like economic and cultural preferences, the model with pro-America MSM endorses more pro-America positions (Figure 2).”

Different generalization, same fine-tuning data
Mind the gap: The claim is demonstrated only with a toy cheese-preference example on one small base model (Llama 3.1-8B) over 4 seeds; broader generality is argued separately via the agentic-misalignment experiments, not this one.
The claim

MSM substantially reduces agentic misalignment.

“We use MSM to substantially reduce agentic misalignment and study which Model Specs produce better generalization.”

The evidence

“Combining MSM with AFT drastically reduces misalignment rates on AM evaluations (Qwen2.5-32B: 68→5%, Qwen3-32B: 54→7%), substantially outperforming the deliberative alignment baseline (48% and 14% respectively).”

Reducing agentic misalignment
The claim

MSM might not scale well with high-compute reasoning post-training.

“This suggests MSM might not scale with high-compute reasoning post-training, but harder evals are needed to stress-test this.”

The evidence

“the performance of AFT with CoT supervision can converge to MSM + AFT performance at high AFT compute”

How does MSM scale with AFT compute?
Mind the gap: The authors' own hedge ('harder evals are needed') flags that the observed convergence could just reflect the eval saturating near zero misalignment rather than genuine parity in capability; the paper does not rule this out.
The claim

Understanding the spec (via MSM) and demonstrating aligned behavior (via AFT) are complementary, not substitutes.

“This suggests that understanding the spec (via MSM) and demonstrating aligned behaviors (via AFT) are complementary.”

The evidence

“As Figure 3 shows, neither MSM nor AFT alone comes close to this.”

Reducing agentic misalignment
The claim

Good values and judgment generalize better than rules imposed as unexplained constraints.

“A hypothesis motivating the latter is that good values and judgment can generalize better than rules imposed as unexplained constraints”

The evidence

“We found that explaining the values underlying rules, or adding more detailed subrules to specs, both improve generalization (Figure 5).”

Model Spec science
Mind the gap: The result shows BOTH value-explanations and rule-elaboration improve generalization over a bare-rules spec, which does not clearly adjudicate between the 'values beat rules' hypothesis and the competing 'more explicit coverage helps' hypothesis the paper poses side by side.
Discussion & after

How they frame it, and what they want next

Their framing

The authors present MSM as a new, complementary training stage that fixes a specific failure of alignment fine-tuning -- that demonstration data underspecifies the intended generalization -- and position it both as a practical safety intervention and as a new empirical methodology for comparing candidate Model Specs before committing to one.

Register: Confident and declarative about their own measured results ('drastically reduces', 'substantially outperforming'), but explicitly hedged ('suggests', 'might') whenever the claim extrapolates beyond the settings actually tested.

Where they hedge

“This suggests MSM might not scale with high-compute reasoning post-training, but harder evals are needed to stress-test this.”How does MSM scale with AFT compute?
“One caveat is that the performance of AFT with CoT supervision can converge to MSM + AFT performance at high AFT compute.”How does MSM scale with AFT compute?

What they say it means

  • MSM could let labs rely less on chain-of-thought supervision during alignment training, which matters for keeping CoT usable as an external monitoring signal.
    the paper’s words
    “This shows that stacking MSM with AFT can teach models aligned reasoning without directly training on CoT, which might be relevant for preserving CoT monitorability ( Korbak et al., 2025 ).”Reducing agentic misalignment
  • Design choices between rule-based and values-based specs, previously settled mainly by philosophical argument, can now be compared empirically.
    the paper’s words
    “important decisions—like whether to rely more on explanations that cultivate holistic judgment or on explicit behavioral rules—have largely been settled through philosophical argument.”Model Spec science

What they call for next

  • Read the full paper for the complete methodology and results.
    the paper’s words
    “To learn more, read our paper .”Conclusion

Limitations they state

“The cheese example is a toy demonstration.”Reducing agentic misalignment
“One caveat is that the performance of AFT with CoT supervision can converge to MSM + AFT performance at high AFT compute.”How does MSM scale with AFT compute?
“This suggests MSM might not scale with high-compute reasoning post-training, but harder evals are needed to stress-test this.”How does MSM scale with AFT compute?
For your own writing

Moves worth stealing

Opens the technical demonstration with a deliberately low-stakes, toy example (cheese preferences) to isolate the core mechanism before moving to the safety-critical result.

“Suppose we want a model to learn pro-affordability values—preferring things that are affordable and accessible over things that are expensive and scarce.”

States a competing hypothesis to its own methodological lineage before testing empirically, rather than only reporting the result the authors expected.

“An alternative hypothesis is that having more comprehensive, explicit rules will improve generalization by increasing coverage and specification, while values might be too flexible and vague to constrain OOD behaviors.”

Cites concurrent red-teaming findings inline with author-year references rather than a separate literature-review section.

“For example, LLM agents have been shown to take unethical actions (e.g., blackmailing, leaking company information, alignment faking) when placed in scenarios different from those appearing in their alignment training”

Closes with a short, itemized list of contributions instead of a discursive conclusion, making the paper's core claims easy to extract and cite.

“We introduce model spec midtraining (MSM),  a method for shaping how models generalize from AFT, by first training them on documents that discuss Model Spec content.”
Connected

Where else this leads

Same people

Same territory

Published alongside it

The nearest publications in time, across all three labs.

What this page was built from

Working from the lab's blog-post announcement (text_grade: partial per corpus manifest), not the full paper PDF; the post has a tl;dr but no section labeled 'Abstract', so abstract is null. The corpus manifest lists pub_date as 2026-05-15 (month precision), but the post's own byline states 'May 5, 2026' explicitly -- both are noted here. The post links to a full PDF ('Paper') but the .txt extraction did not capture that link's URL, so pdf_or_paper_url falls back to the blog's own source_url.