Your AI guardrail is green. It's also catching nothing.

There are three ways a security guardrail can fail you.

Two of them are loud. One of them is a serial killer.

1. It's down. Crashed, misconfigured, not deployed. You find out fast — errors, alerts, a red dashboard, an on-call page at 3am. Painful, but honest.
2. It's weak. It runs, it catches some attacks, it misses others. You can measure it, argue about it, improve it. Also honest.
3. It's running perfectly, passing every health check, returning valid scores on every request — and configured to catch nothing. No error. No alert. No red anything. Every dashboard it touches is green by construction.

The third one is the problem. Because it has no symptom. Your monitoring says "guardrail: healthy ✅" and it's telling the truth. The guardrail is healthy. It's just not a guardrail.

I ran into a textbook case of #3 while benchmarking prompt-injection detectors, and it's worth dissecting, because once you see the shape of it you'll start finding it everywhere.

The setup: the smoke alarm of the AI stack

Everyone shipping an AI agent bolts on a prompt-injection detector — a little classifier that reads the text flowing into the model and screams if it smells an attack. It's the smoke alarm of the agent world. You install it, you see the green light, you move on.

So I did the obvious experiment: I took 10 of these smoke alarms and set 629 real fires.

Specifically — I took 629 real injection attacks from AgentDojo, buried each one inside ordinary tool output (a bill, an email, a web page — the way an agent firewall actually sees them, not the clean lab version), and ran 10 open-source detectors over the lot, plus 97 benign tool outputs to catch the ones that just alarm at everything.

Most of the results were the boring kind of bad (the "weak" and "screams at toast" failures). But one detector produced the interesting kind of bad.

Exhibit A: Meta's Prompt Guard 2, catching 1%

Meta's Prompt Guard 2 — the model everyone name-drops — caught 6 of 629 buried attacks. About 1%.

Now, your first instinct is "the model is bad." Reasonable instinct. Wrong.

Here's what the model was actually doing on each request:

Look at the top row. The attack scores roughly 10x higher than normal text — the model's ranking is nearly perfect. As a discriminator, it works.

Now look at the cutoff. Use the model the obvious way — the standard 0.5 decision boundary you'd apply to any binary classifier out of the box (block if P(malicious) ≥ 0.5) — and every score it produces, the attack at 0.009 and grandma at 0.0008 alike, sits far below the line. So the verdict on every single request is identical: "nah, we're good." ✅

(To be precise, because this is the sentence people will poke at: 0.5 isn't some evil value Meta hard-coded — it's the threshold you get by treating a probability classifier the normal way. The failure isn't a bad config someone shipped; it's that **the obvious, default way to use this model is off by ~50x for buried attacks* — and nothing about the running system would ever tell you.)*

The detector was producing different scores. The control just wasn't doing anything with the difference.

It's a smoke alarm with the sensitivity dial turned all the way to "only trigger for a literal supernova." The sensor is fine. The wiring is fine. It will faithfully detect the heat death of the universe. Your kitchen fire? Not so much.

That is not a weak guardrail. That is a perfectly functional discriminator thresholded into a no-op. It loads, it returns scores, it passes health checks, it logs clean — and it catches 1% of attacks. Green by construction.

The plot twist that makes it worse

Here's the part that should genuinely unsettle you.

I re-ran Prompt Guard 2 with the threshold moved from 0.5 down to 0.003 — into the range where its scores actually live — and calibrated so it wrongly flags at most 2% of normal traffic, measured on an AgentDojo domain it was never tuned on.

It went from catching 1% to catching 99% (621/629).

Same model. Same weights. Same requests. One number. The difference between "protects you from basically nothing" and "catches almost everything" was a config value the model shipped with — off by roughly two orders of magnitude for this use case.

The uncomfortable takeaway isn't "the model is good" or "the model is bad." It's that the shipped default was wrong by ~100x, and nothing about the running system would ever tell you. Every health check passes at 1% exactly as it does at 99%.

(Honesty clause, because this is the part everyone screenshots wrong: don't read "99%" as "Prompt Guard 2 solves prompt injection." Every AgentDojo attack uses the same wrapper template, so a threshold tuned that finely may be recognizing the template, not attacks in general. The load-bearing claim is the miscalibration — "the default is wrong by 100x" — not the 99%. Tune on **your* traffic, then trust a number.)*

Why "green by construction" is the dangerous class

Compare the three failure modes by how you'd find out:

The first two fail toward visibility. The third fails toward silence. It's the security equivalent of a bouncer who's clocked in, in uniform, standing at the door, checking every ID — with his eyes closed. Every metric you have says "door: staffed." The metric you don't have says "staffed ≠ guarding."

And "it passed the health check" is doing a lot of unearned work in most teams' heads. This is the gap between a liveness check (is the thing running?) and an effectiveness check (is it actually stopping what it's supposed to stop?). A health check proves liveness. It says nothing about effectiveness — and almost every dashboard measures the first while quietly implying it measured the second.

How to actually tell if yours is armed

This isn't a "shame on Meta" post — the model does what it was trained to do, and defaults have to be conservative to avoid blocking everyone. It's a "shame on us if we trust the green light" post. Concretely:

1. A score that ranks well with a bad cutoff is worth zero. Separately measure discrimination (does it rank attacks above benign?) and operating point (is the threshold where the scores actually live?). A model can be great at the first and useless at the second — which is exactly this case.
2. Never trust a vendor's default threshold on your data. It was picked for a distribution that isn't yours. Sweep it. Find where your attack scores and your benign scores actually sit.
3. Measure at a false-alarm budget, not in the abstract. "Catches 99%" is meaningless without "…while wrongly blocking X% of normal traffic." A detector that blocks 98% of legit calls isn't a control; it's a way to make everyone route around your agent. (Two detectors in my benchmark did exactly this and still "scored" 100% on attacks. A brick over the deny button scores 100% too.)
4. Health check ≠ armed check. Add a test that fires a known attack through the live guardrail and asserts it's blocked. If your monitoring can't tell the difference between "catching 99%" and "catching 1%," your monitoring is green by construction too.
5. Red-team the config, not just the model. The vulnerability here wasn't in the weights. It was in a single threshold. Your attack surface includes your YAML.

Or, as a checklist you can staple to any guardrail:

Liveness check:     Is the detector running at all?
Effectiveness check: Does it BLOCK a known attack, right now, in prod?
Calibration check:   Are attack scores actually separated from benign scores?
Operating-point check: At our tolerable false-alarm rate, what % of attacks do we catch?
Regression check:    Is all of the above still true after the next model/config change?

Only the first line is on most dashboards. The other four are the difference between a control and a decoration.

The bigger, more uncomfortable question

Prompt Guard 2 at 1% is a clean, measurable example because I had 629 attacks to throw at it. But the shape generalizes way past prompt injection:

• The WAF rule set that's deployed but in "log-only" mode.
• The alert that's been firing into a muted channel since Q1.
• The MFA that's enforced… except for the legacy API path.
• The test suite that's green because half of it is skip.

All green. All healthy. All protecting you from a supernova and nothing smaller.

So the question I'd actually sit with: how many of your green dashboards are green because the thing works — and how many are green by construction? The scary answer is that, by definition, you can't tell the difference by looking at the dashboard. You have to fire a real attack at it and watch.

The whole benchmark — 10 detectors, 629 attacks, the threshold sweep, the cross-domain calibration — is open and reproducible here, if you want to see which alarms are real and which are decorative:

github.com/rudratoshs/buried-injections ⭐

The full thing — 10 detectors, 629 attacks, the threshold sweep, the cross-domain calibration — is open and reproducible. If you find a flaw in the methodology, I genuinely want to know — security benchmarks are more useful when people try to break them. And if it made you want to go check whether your own guardrail is armed or just green, a ⭐ helps it reach the next person about to trust a green light. 🙏

Be honest in the comments: have you ever found a security control that was "on," passing every check, and doing absolutely nothing? What was it, and how did you finally catch it — an incident, a red-team, or dumb luck? 👇

I write about AI, security, and the honest ways things break — benchmarks with the false-positive column left in. Follow me here if that's your lane. 👋