SLEIGHT-Bench: Finding Blind Spots in AI Monitors
Anthropic and collaborators built SLEIGHT-Bench, 40 synthetic transcripts exploiting known blind spots in AI monitors, and found frontier monitors miss most of the attacks.
It maps concrete, exploitable weaknesses in the LLM-based monitoring systems that AI labs are relying on to catch misaligned agent behavior.
Elle Najt · Colin Toft · Tyler Tracy · Fabien Roger · Joe Benton — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
This source carries no verbatim abstract.
The authors built a benchmark of 40 fake but realistic attack transcripts, each designed to slip past AI monitors by exploiting a specific known weakness, like monitors being bad at counting or losing track of file contents. When they tested frontier models (Claude, Gemini, GPT) as monitors on these transcripts, most attacks went undetected, though targeted prompt changes helped catch more of the attacks they were aimed at.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
Monitor blind spot
“each attack exploits what we call a monitor blind spot”SLEIGHT-Bench
In plain terms: A specific capability gap in a monitor model that makes it systematically miss certain kinds of harmful actions.
SLEIGHT-Bench
“we present SLEIGHT-Bench (Subtle Low-itEration Insight-Guided Harmful Transcripts), a benchmark of synthetic transcripts containing 40 attacks across 11 categories, each showing a coding agent covertly pursuing a harmful objective that frontier monitors mostly can't detect.”SLEIGHT-Bench
In plain terms: The paper's benchmark: 40 fabricated coding-agent transcripts, each hiding a harmful action in a way designed to exploit a monitor's blind spot.
AI control
“such monitoring could help ensure AI control , that is preventing potentially misaligned AIs from causing a catastrophe even if they strategically try to cause one.”Background
In plain terms: Safety measures aimed at stopping a misaligned AI from causing serious harm even if it is actively trying to.
1% false positive rate (FPR) calibration
“We calibrate our monitors to a 1% false positive rate, meaning they would only incorrectly flag 1 in 100 benign transcripts, and measure how many real attacks they catch at that threshold.”Evaluating monitors
In plain terms: A monitor's alert threshold is set so it only wrongly flags 1 in every 100 harmless transcripts, then its catch rate on real attacks is measured at that setting.
Any-trial catch rate
“The any-trial rate counts an attack as caught if at least one of 10 trials scores above the threshold.”Results
In plain terms: An attack counts as detected if the monitor flags it in at least one out of ten repeated attempts.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Researchers manually identify a conceptual blind spot in how frontier monitors reason, rather than letting an AI discover attacks on its own.
Trace this step to the paper
“identify a conceptual blind spot in how monitors reason”SLEIGHT-Bench
- AI assistance is used to write out a realistic transcript that exploits the identified blind spot while looking benign.
Trace this step to the paper
“use AI assistance to generate a transcript that exploits the blind spot to appear benign”SLEIGHT-Bench
- Each finished transcript is evaluated against frontier monitor models.
Trace this step to the paper
“evaluate against frontier monitors”SLEIGHT-Bench
- Automated checks confirm each synthetic transcript is internally consistent, that the attack would really succeed, and that the incriminating information is present (even if hidden).
Trace this step to the paper
“we run automated checks to verify that each transcript is internally consistent, that the attack would actually succeed, and that the information necessary to flag the attack is present, even if hidden.”SLEIGHT-Bench
- A matched benign version of each attack transcript is created, with the harmful action removed, to calibrate what counts as a false positive.
Trace this step to the paper
“We calibrate the monitor threshold by developing benign variations of the attack transcripts, where the malicious action is removed and the transcript is modified to match the user’s instructed behavior.”Evaluating monitors
- Each monitor's alert threshold is set so it only misfires on 1% of benign transcripts, and its catch rate is measured at that threshold.
Trace this step to the paper
“We calibrate our monitors to a 1% false positive rate, meaning they would only incorrectly flag 1 in 100 benign transcripts, and measure how many real attacks they catch at that threshold.”Evaluating monitors
- All 40 attacks are run against frontier models from multiple providers acting as monitors.
Trace this step to the paper
“We test our 40 attacks against frontier models from multiple providers, including Claude, Gemini, and GPT models, to see how they perform as monitors.”Evaluating monitors
- Each monitor is given the transcript and a harms-focused prompt, then asked to output a 0-100 suspiciousness score.
Trace this step to the paper
“then ask them to rate the suspiciousness of the agent's actions on a scale from 0-100.”Evaluating monitors
- For Claude models, the monitor is also run as an active coding agent (using Claude Code) that can execute commands, rather than just passively reading the transcript.
Trace this step to the paper
“For Claude models, we also test running the monitor as a coding agent using Claude Code, rather than just having it read the transcript passively.”Evaluating monitors
- The dataset is re-tested with monitor prompts specifically engineered to target individual blind spots (e.g. state tracking, authorization confusion, environment profiling).
Trace this step to the paper
“We also test our dataset on monitor prompts targeted at specific blind spots.”Evaluating monitors
Exactly what was run, and how
| Model | Developer | Temp | Effort / reasoning | Deployment | Other settings |
|---|---|---|---|---|---|
| Opus 4.6 | Anthropic | not reported | not reported | unstated | — |
| Claude models (unspecified versions) | Anthropic | not reported | not reported | unstated | — |
| Gemini models (unspecified versions) | Google DeepMind | not reported | not reported | unstated | — |
| GPT models (unspecified versions) | OpenAI | not reported | not reported | unstated | — |
Source for Opus 4.6 settings
“On Opus 4.6, the main monitor we work with, 50% of attacks are never caught on any of the 10 trials.”Results
Source for Claude models (unspecified versions) settings
“We test our 40 attacks against frontier models from multiple providers, including Claude, Gemini, and GPT models, to see how they perform as monitors.”Evaluating monitors
Source for Gemini models (unspecified versions) settings
“We test our 40 attacks against frontier models from multiple providers, including Claude, Gemini, and GPT models, to see how they perform as monitors.”Evaluating monitors
Source for GPT models (unspecified versions) settings
“We test our 40 attacks against frontier models from multiple providers, including Claude, Gemini, and GPT models, to see how they perform as monitors.”Evaluating monitors
What they reported — and what they left out
The post names Opus 4.6 as its primary monitor and says it also tested unspecified Claude, Gemini, and GPT models, extended-thinking mode, and a Claude-Code coding-agent variant, but it never states exact model versions, temperatures, sampling settings, or system prompts beyond referencing 'the ControlArena-adapted prompt' and an unquoted internal Anthropic prompt.
The numbers they report
The benchmark covers 40 attacks spanning 11 blind-spot categories.
40 attacks, 11 categories
See it in the paper
“a benchmark of synthetic transcripts containing 40 attacks across 11 categories”SLEIGHT-Bench
File/object reuse attacks (monitor loses track of overwritten file/variable contents) are caught rarely.
15%
See it in the paper
“Monitors lose track of what a file or variable contains after it's been overwritten or reassigned”Our list of blind spots
Counting-based attacks (monitor can't reliably count lines/offsets/items) are caught rarely.
13%
See it in the paper
“Monitors can't reliably count lines, offsets, or items in long outputs”Our list of blind spots
System-state attacks (monitor doesn't track cumulative system changes) are caught less than half the time.
46%
See it in the paper
“Monitors don't track cumulative changes to the system, like disk usage or background processes”Our list of blind spots
Model-priors attacks (monitor assumes a standard environment configuration) are caught rarely.
13%
See it in the paper
“Monitors assume a standard environment configuration (e.g. English, Linux, Bash) and fail to notice when the system is different”Our list of blind spots
Authorization-confusion attacks are caught more often than most other categories.
55%
See it in the paper
“Monitors struggle to distinguish genuinely user-directed actions from actions the agent has manufactured justification for”Our list of blind spots
Needle-in-haystack attacks (malicious content hidden in opaque long files) are caught well under half the time.
38%
See it in the paper
“Monitors miss malicious content embedded within long files, particularly opaque data formats”Our list of blind spots
N-hops attacks (chaining facts to infer a harmful outcome) are never caught.
0%
See it in the paper
“Monitors don't reliably chain multiple calculations or facts together to determine that an action would produce a harmful outcome”Our list of blind spots
Plausible-naming attacks are caught at a high rate, though the authors note these fall right at the FPR threshold.
90%
See it in the paper
“Monitors sometimes believe that realistic-sounding bucket names, URLs, etc. are legitimate (the catch rate here is high but these transcripts fall right above the 1% FPR threshold we tuned)”Our list of blind spots
Omission attacks (harm from what the agent didn't do) are never caught.
0%
See it in the paper
“Monitors are less likely to notice something the agent didn't do than something it did do”Our list of blind spots
Multi-session attacks (split across separately monitored sessions) are caught less than a third of the time.
30%
See it in the paper
“When an attack is split across separately monitored sessions, each half appears innocent on its own, and the changes the first makes are not as salient, even if they are visible”Our list of blind spots
Jailbreak-style attacks (confusing the monitor's own instructions with transcript content) are never caught.
0%
See it in the paper
“Monitors struggle to distinguish their own instructions from the transcript content they're evaluating”Our list of blind spots
On the primary monitor, half of all attacks evade detection across all 10 trials, and only a fifth are reliably caught.
50% never caught in 10/10 trials; 8/40 reliably detected
See it in the paper
“On Opus 4.6, the main monitor we work with, 50% of attacks are never caught on any of the 10 trials.”Results
Running the monitor as an active coding agent is far more computationally expensive than a passive API call.
~10x more model invocations, ~14x more output tokens
See it in the paper
“The Claude Code monitor on average takes ~10x more model invocations and generates ~14x more output tokens than the API call”Evaluating monitors
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
Frontier monitors currently detect the benchmark's attacks at a low rate overall.
“Detection rates are generally low.”
“On Opus 4.6, the main monitor we work with, 50% of attacks are never caught on any of the 10 trials.”
ResultsAttacks that exploit monitor blind spots reliably evade detection, showing monitoring systems aren't yet robust to sophisticated adversaries.
“But our dataset shows that attacks targeting monitor blind spots can reliably evade detection, and monitoring systems are not yet robust to more sophisticated adversaries.”
“Only 8 out of 40 are reliably detected.”
ResultsBlind-spot-targeted monitor prompts improve detection on the categories they target but can hurt detection elsewhere.
“we find these prompts improve detection on the categories they target, but degrade performance on at least one other.”
“the state-tracking prompt catches more file-reuse attacks, but misses more authorization-confusion attacks.”
Evaluating monitorsAcross the models tested as monitors, more compute generally produces better detection.
“the general trend shows that more compute means better detection.”
“While we always let monitors write out their reasoning before arriving at their score, enabling extended thinking as well improves performance for most models.”
Evaluating monitorsLetting a monitor execute code (rather than just read the transcript) helps it verify an agent's claims.
“Being able to execute code helps the monitor verify claims the agent makes”
“for example, if an agent copies a large file but subtly alters part of it, a coding agent monitor can write a script to compare the two, while a regular monitor would have to take the output at face value.”
Evaluating monitorsHow they frame it, and what they want next
Their framing
The authors frame SLEIGHT-Bench as mapping concrete monitor weaknesses so defenders have a list to patch, positioning LLM monitoring as one layer within a broader defense-in-depth safety strategy rather than a complete solution. They are careful to frame their attacks as researcher-designed illustrations of possible failure modes, not evidence that today's models can independently find or execute such attacks, and they explicitly weigh the dual-use risk of releasing the benchmark against its value for community-driven improvement.
Register: Measured and hedged throughout: findings are qualified with phrases like 'we believe,' generalizations from the benchmark to real-world adversaries are explicitly disclaimed, and the dual-use risk of the release is acknowledged rather than downplayed.
Where they hedge
“We believe current frontier models would not independently discover and execute attacks like ours.”Results
“While this cost might be high, it may be worth it in high-stakes applications where misaligned actions could increase the chance of existential catastrophes, and it may also be mitigated”Evaluating monitors
“This work is dual-use: attack techniques could uplift misaligned AI if scraped and trained on, but we believe public release serves safety on balance”Responsible Disclosure
What they say it means
- Since no single monitor prompt covers every blind spot, combining several specialized prompts into an ensemble may catch more attack types than any one prompt alone.
the paper’s words
“No single prompt covers everything, suggesting that using an ensemble of multiple specialized prompts, each watching for different blind spots, may be a promising path forward.”Evaluating monitors
- The blind-spot taxonomy could inform automated red-teaming that tests real agents in real environments, beyond this benchmark's synthetic transcripts.
the paper’s words
“these blind spots could also guide automatic red-teaming, which tests real agents in real environments, as MonitoringBench”Conclusion
What they call for next
- Invites other researchers to identify additional recurring model failure modes and turn them into new blind-spot transcripts for the benchmark.
the paper’s words
“We're releasing our benchmark and inviting other researchers to help stress-test monitors further — if you've noticed any repeated model failure modes, perhaps they can be framed as blind spots and developed into evasive transcripts!”Conclusion
Limitations they state
“We believe current frontier models would not independently discover and execute attacks like ours.”Results
“it increases the risk of inconsistencies or lack of realism”SLEIGHT-Bench
“Notably, both of these strategies come at a cost.”Evaluating monitors
“the catch rate on model priors attacks might be further improved by a better scaffold and monitor prompt”Evaluating monitors
Moves worth stealing
Gives each failure mode a memorable, reusable label (e.g. 'model priors', 'needle in haystack') so future safety work can slot into the same taxonomy.
“this leads to a blind spot we call "model priors."”
Uses a cute backronym for the benchmark name, making it memorable and easy to cite.
“SLEIGHT-Bench (Subtle Low-itEration Insight-Guided Harmful Transcripts)”
States the entire end-to-end method as a compact numbered list in one sentence before elaborating on each part.
“identify a conceptual blind spot in how monitors reason”
Addresses dual-use and responsible-disclosure concerns head-on in a dedicated section rather than leaving readers to wonder about safety of releasing attack techniques.
“This work is dual-use: attack techniques could uplift misaligned AI if scraped and trained on, but we believe public release serves safety on balance”
Closes with an open invitation for community contribution, turning readers into potential collaborators on the taxonomy.
“We're releasing our benchmark and inviting other researchers to help stress-test monitors further”
Where else this leads
Same people
- Fine-Tuned Lie Detectors Failed to Generalize Anthropic
shares Fabien Roger - Introducing the Conceptual Reasoning Index Anthropic
shares Joe Benton - TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic
shares Joe Benton - Diffuse AI Control on Fuzzy Tasks Anthropic
shares Joe Benton
Same territory
- Patterns and problems in emerging multiagent systems Anthropic
ai safety alignment - Solipsistic superintelligence is unlikely to be cooperative Google DeepMind
ai safety alignment - Realistic honeypot evaluations for scheming propensity Google DeepMind
ai safety alignment - Introducing the Conceptual Reasoning Index Anthropic
ai safety - TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic
ai safety - Training a Misaligned Reward Seeker Anthropic
ai safety
Published alongside it
The nearest publications in time, across all three labs.
- HeadVis Anthropic
2026-05-15 - Model Spec Midtraining: Improving How Alignment Training Generalizes Anthropic
2026-05-15 - Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations Anthropic
2026-05-15 - Teaching Claude Why Anthropic
2026-05-15
What this page was built from
This is Anthropic's alignment-science blog post about SLEIGHT-Bench (manifest text_grade: partial), which links out to 'the full paper' and a dataset repo whose actual URLs are not captured in the extracted plain text; the post has a tl;dr rather than a formally labeled abstract, so abstract is null.