Here is a failure I had not thought about until someone asked me the right question.
You write a PreToolUse hook. It reads the tool call, decides the command is dangerous, and blocks it. You write unit tests for it, because a guard with no tests is a wish. The tests pass. Months go by and the hook never fires, which you read as good news, because nothing bad has happened.
Then one day you find out the harness renamed a field in the payload eighteen weeks ago, your hook has matched nothing since, and every one of your tests is still green. They're green because the test builds the payload itself, in the shape you believed the harness sends. The test and the hook agree with each other perfectly. Neither of them has spoken to the harness in months.
Green tests, dead tripwire. The guard and its tests share a belief, so the tests cannot see the belief go stale.
This is the same shape as a bug I wrote about last time, where a redaction tool verified its own output using the same wrong geometry it had used to draw with. Different domain, identical structure: a check built from the same assumption as the thing it checks is not a check, it's a second copy of your belief.
Why "it hasn't fired" tells you nothing
A guard has two states that look identical from the outside. Working and quiet, because nothing tripped it. Broken and quiet, because it cannot match anything any more.
Nothing in your workflow distinguishes them. You don't get a weekly report of hooks that ran. The absence of refusals feels like safety, and in the broken case it is precisely the opposite: you've been running unguarded while believing you were covered, which is worse than having no hook at all, because you'd have been more careful without one.
Every guard that encodes a fact about an external world has this problem. The payload shape of your agent harness. The name of a tool. Whether CI still runs the check your hook assumes runs. Those facts change without telling you, and the guard goes quiet rather than loud.
Fail closed, but not everywhere
The obvious fix is to fail closed. If this is a tool I'm guarding and the field I need isn't there, that's an anomaly, not an allow. Most hooks, mine included, default to allowing anything they can't parse, which is exactly backwards.
The question I got asked, and did not have a good answer for at first, was whether that means failing closed everywhere, or only for guards on irreversible things.
I now think irreversibility is the wrong axis. The right one is what happens when the hook declines to decide. Three modes, three correct failures:
Block mode: fail closed. The hook is certain from the input alone that the call is wrong. If it can't read the input, refuse. The fallback is a retry or a question, both cheap, and the false block you eat on the day the harness changes is the thing that tells you the shape moved. That one annoying refusal is the monitoring you never set up.
Warn mode: don't fail closed, and say why. A warn hook exists because the rule is a heuristic that will fire on legitimate work, so turning an unparseable payload into a wall contradicts its own design. Emit the warning anyway. And the detail I'd have got wrong: the warning has to say precisely that it couldn't read the input, rather than printing its usual generic text. A warning that can't name the file or the command, but talks as if it can, teaches the model to discount every future warning from that hook.
Auto-allow mode: already fails closed, for free. This is the mode nobody discusses. Hooks that pre-approve read-only commands so they never surface a permission prompt. If one can't parse the input, it simply doesn't allow, and you land on the normal prompt. The fallback is a human, which is exactly where you want to be.
So the rule I'd write now: fail closed wherever declining costs a prompt. Don't, wherever declining costs a block on legitimate work.
The fixture that lies to you
The other half is contract tests, and there's a trap in them.
Recording one real payload from an actual session and replaying it against the hook catches the rename before it costs you anything. Writing that fixture by hand does not, because a hand-written fixture is your belief about the payload shape stored in a second file. You now have two copies of the same assumption and a false sense of coverage. If you can't capture a real one, log the raw input on every invocation and go read it occasionally. Dull, works.
My own setup, honestly: two hooks have real unit tests, one with a table of about thirty adversarial command variants, and both assert the deny path rather than only the happy one. Neither has a recorded fixture. I haven't fixed it yet.
What I still don't have
The thing I'd actually want is a canary. A scheduled job that deliberately issues a known-bad call and asserts it gets refused, so a dead guard announces itself instead of waiting to be discovered. That's a cron and an assertion, it isn't hard, and I haven't built it, which I suspect means most people running a dozen hooks haven't either.
Most of this post came out of the comments on my last one rather than from me. @build996 pushed on which refusals an agent can actually act on, @max_quimby on hooks quietly becoming an unversioned rulebook nobody remembers the reasons for, @syntaxwanderer_26 asked the question that produced the whole fail-closed taxonomy and then supplied the warn-mode detail, and @doriansotpyrc asked how a rule earns promotion from prose into a hook, which I answered as: once means write the sentence, twice means the sentence isn't carrying it.
So, the part I'm stuck on. How would you detect a guard that has been dead for a month? A canary is the only honest answer I've got, and it only covers the failures you thought to simulate. If you've found something better, or you've had a tripwire die quietly and only noticed by accident, I'd like to hear it.
Top comments (10)
Good to see the thread grow into a post. The canary is the piece we never got to in the comments.
On detecting a guard that's been dead for a month, I'd count rather than simulate. A canary proves the guard still refuses the one call you scripted. It can't see the guard quietly losing its grip on real traffic. And logging inside the hook isn't enough either: if the matcher stops matching, the hook is never called and logs nothing, which looks exactly like a quiet week. So compare what the hook says it saw with a record the hook doesn't produce. If the remote shows twelve pushes this week and the push guard saw zero, it's dead, and you found out without guessing what the failure would look like.
It's your redaction bug turned around: the check has to come from a source that doesn't share the guard's belief. The VCS, CI logs and the harness transcript qualify. The hook's own fixtures don't.
Semitexa's
ai:verifyfollows the same rule for its own empty runs: when no check matches the change, the verdict isnothing_to_verify, not a pass, because "nothing ran" and "everything passed" must never look the same.Is there anything in your harness today that would give you that independent count, or would you have to build it?
Counting rather than simulating is better than what I had, and the reason is the part I'd missed: a canary only proves the guard still refuses the call I thought to script.
To your question, it turns out I already have the independent record and hadn't noticed it. The harness writes a JSONL transcript per session, and every tool call lands in it with its name and input, written by the harness rather than by any hook. I counted one of mine just now: 507 Bash calls, 328 edits, 186 writes. Hook refusals show up in there too, so both halves of your comparison come from the one source that doesn't share the guard's belief.
What makes that better than instrumenting forward is that it's retrospective. There are 594 of those transcripts sitting on my disk. So I can ask today whether the pooler guard has seen a matching command in the last month and find out it died in September, instead of adding a counter and waiting a month to learn the same thing. The audit I thought I'd have to wait for is already in the logs.
And agreed on the empty-run verdict. "Nothing ran" and "everything passed" being indistinguishable is the whole post.
And thank you for the comment, this is the most insightful comment I've had on this site.
Thank you, that means a lot from the person who wrote the post. The transcripts are a better answer than mine: written by the harness, retrospective, and already on disk, so the audit costs one script instead of a month of waiting. I'm curious what the count says for your pooler guard. And your thread already changed something on our side: today I filed a task to move our ai:verify gate out of the instructions file and into a Stop hook, with your "fail closed only where declining costs a prompt" rule and version-stamped recorded fixtures written into it.
The fix that keeps working after the harness changes again: stop building the payload in the test by hand. Record it. Capture the last dozen real PreToolUse calls your harness actually makes — they're small, and nothing sensitive lives in the shape — commit them as golden files, and have the unit test assert the hook matches at least one recent real payload.
Then a harness field rename turns your CI red within a day instead of leaving the tripwire dead for eighteen weeks. The test stops being a mirror of your belief and becomes a diff against the outside world, which is the thing that actually changed.
This composes with the canary idea rather than replacing it: the canary proves the hook still fires at runtime, golden payloads prove the matching logic still describes reality. You want both, because they go quiet in different ways.
Same as a smoke detector nobody tests. It stays quiet for months and everyone reads quiet as safe. Pushing the button is the only way to know. We're building that kind of real-world check into ML Systems.
A harness renamed one field and the tests stayed green because the test built its own payload. A check I'd add is a canary run: on a schedule, feed the hook a payload captured from the real harness rather than one written by hand, and require that a known-dangerous command is actually blocked. I'd also keep a counter of how often the hook matched anything during normal use and alert when it flatlines. That way a quiet hook and a broken hook stop looking identical, and the unit tests no longer have to carry that job alone.
The 'two states that look identical from the outside' line is the whole problem — a quiet guard reads as a working guard, and the broken case is worse than no guard because you'd have been more careful without one. Your warn-mode detail is the one I'd steal: a warning that can't name the file or command but talks as if it can teaches the model to discount every future warning. On the canary — have you tried making the scheduled bad call part of the hook's contract test, or does that just move the staleness problem one layer up?
One of my gates passed whenever its log file was missing. It enforced the hardest rule in my agent test harness, no live API calls outside the planned captures, by reading the call log. If the file didn't exist it returned True with "absent, so zero calls logged", so a recorder that wrote nothing would have passed. An agent found it while reading the gate code. The fix makes a zero count only when a non-zero next to it shows the recorder ran.
Since the transcripts are already on disk, I'd run the count per hook per week instead of one total. A hook whose matches drop to zero in a single week points straight at the harness update that renamed the field, which matters when the break is eighteen weeks old.