Every AI-and-engineering post right now is about adding a guardrail.
Lint rules the agent can't bypass. Evals before you ship a prompt change. A review bot on every PR. A regression suite built from real production failures. Architecture rules checked into the repo so the model reads them on the way in.
I have written some of those posts. I still believe in them. This is the work that makes AI usable by senior engineers instead of a liability.
But I have been in three separate conversations on this platform in the last week that were all, underneath, about the same thing, and it isn't about which guardrails to add.
A guardrail that has never fired and a guardrail that silently stopped running produce identical output.
Green.
The pipeline that had never been green
Vicente Reyes had a GitHub Actions workflow called Deploy to DigitalOcean. Fully wired. SSH action, secrets, the works. And every time he shipped a backend change he still SSH'd into the droplet and ran git pull by hand.
The deploy job was gated on CI passing.
on:
workflow_run:
workflows: ['CI']
branches: ['main']
types: [completed]
jobs:
deploy:
if: ${{ github.event.workflow_run.conclusion == 'success' }}
Sensible enough. The problem was that CI had never once gone green on main. Not flaky. Never. So every deploy run showed skipped, forever, and the pipeline sat permanently behind a gate that could not open.
His line about it is the one worth keeping. A pipeline that is silently and permanently blocked looks, from a distance, exactly like a pipeline that doesn't exist.
Nothing in the Actions UI says this workflow has not succeeded in forty runs. You have to go and ask.
The grader that can't say no
On a thread about designing trustworthy AI evals, Heinrich Neb made the point that every grader needs a known-bad twin. An input it is supposed to reject, plus a recorded date of when it last actually rejected something.
His framing is the sharpest version of this I have seen: a grader that has never failed and a grader that silently stopped running print the same green.
Same failure as Vicente's pipeline, one layer up. In his case the gate was stuck closed. In an eval suite the gate is stuck open, which is worse, because a stuck-closed gate is annoying enough that somebody eventually investigates. A stuck-open gate just keeps saying yes.
Think about how an eval suite actually rots. Somebody changes a prompt template and the grader's regex stops matching, so everything scores as pass. Somebody renames a dataset field, the loader returns an empty list, the suite runs zero cases in 0.4 seconds and reports 100%. A provider changes a default and your grader model gets more agreeable.
All three look like success.
The score with no provenance
The third one is mine, from the same thread.
An eval result with no harness version, no dataset snapshot and no prompt revision attached to it is not evidence. It is a self-reported claim.
Which is fine right up until the number moves. Then somebody asks whether the model got better or the suite got easier, and if you can't answer, you never had a measurement. You had a vibe with a decimal point on it.
That reflex comes from working in payments. In a regulated system nobody asks you to trust that a control ran. They ask you to demonstrate which control ran, on what input, at what time, under which version of the rules. Months later. To somebody who wasn't there and isn't inclined to take your word for it.
Engineering has quietly inherited that burden. Evals are increasingly the artefact a shipping decision rests on. They just haven't inherited the paperwork.
What actually ties these together
When you automate a check, you swap one question for another and don't notice.
Before automation the question is did someone look at this? You know the answer, because you can see the person and ask them.
After automation the question you think you are asking is still did the check pass? But the question you are now depending on is is the check alive?
Almost nobody instruments the second one.
This is well understood in operations. You don't only alert on errors, you alert on the absence of a heartbeat, because a monitoring system that dies looks exactly like a system with no problems. Dead man's switches exist for precisely this reason.
We have somehow not carried it across to the checks that gate our code. A CI workflow, a lint rule, an eval suite, an agent policy. These are all monitoring systems for correctness, and we run them with no heartbeat at all.
What a guardrail needs before you trust it
None of this is exotic.
A known-bad input it must reject. Every check needs a case it is supposed to fail on, running alongside the real ones. If your lint rule can't catch its own canary, the lint rule isn't running. If your eval's negative control scores as a pass, the grader is broken and every other number in that run is noise. Same idea as a negative control in a lab. Nobody trusts an assay that only ever comes back clean.
A recorded date of last rejection. Not when it last ran. When it last said no. A guardrail that hasn't rejected anything in four months is either protecting an unusually disciplined team or it broke in May, and those look identical on a dashboard. Put it in a column somewhere. If nobody can answer "when did this last catch something", it is decoration.
A run count somebody occasionally looks at. The zero-cases failure is the sneakiest one, because a suite that loads an empty dataset passes fast with a perfect score. Assert on the count. If it expects 240 cases and got 0, that is a hard failure, not a 100%.
Provenance on the result. Harness commit, dataset hash, prompt revision, model version, timestamp. Attached to the score, not sitting in a CI log with thirty-day retention. The test is simple. Six months from now, can you reproduce this exact number? If not, you can't use it to defend a decision, which means it was never really the reason for the decision.
Why this gets worse with agents, not better
The reason I keep coming back to this is that AI shifts the ratio.
The argument for agents in engineering, the one I actually believe, is that you get something close to an army of near-zero-mistake juniors. The constraint stops being how fast people can write code and becomes how fast people can review it. So you compensate by pushing more review into automation. More lint rules, more tests, more evals, more policy checks.
That is the right move. I would make it again.
But it means the fraction of your correctness resting on unattended machinery goes up sharply. When a human reviewed everything, a broken lint rule was a small hole in a large net. When automation reviews everything, the broken lint rule is the net.
The guardrails become load-bearing at exactly the moment nobody is watching them closely enough to notice they stopped.
Go and check one
Pick the guardrail you would be most upset to lose. The eval suite that gates prompt changes, the rule that stops an agent touching the payment path, whatever yours is.
Then answer one question about it. When did it last say no?
If you can find that out in under a minute, good. If you can't find it at all, you don't have a guardrail. You have a green light with nothing behind it, and you have been treating it as evidence.
Credit where it's due. The "never failed and stopped running print the same green" framing is Heinrich Neb's, and the permanently-gated pipeline is Vicente G. Reyes'. I just noticed they were the same bug.
I'm Arun, CTO and co-founder at Atoa. We build open banking payments for the UK. I write about AI, payments, and the messy parts of running engineering systems. @mickyarun.
Top comments (20)
The negative control canary is the only pattern that reliably catches parser drift in policy filters. When an agent runner updates its tool call schema or changes how whitespace is stripped from shell arguments, argument-matching regexes often stop matching the payload structure entirely. Because the parser sees no blacklist hits, every command passes through.
I ran into this after a runtime update changed JSON serialization on bash arguments. The filter silently evaluated every destructive command as benign because the regex was looking for a string pattern that no longer appeared in the raw input. The CI suite stayed green because nothing failed explicitly.
The fix that held up was bundling a synthetic poison payload into every filter test run. If the gate fails to reject the known bad command, the test harness hard-errors immediately. Treating a guardrail that never rejects anything as a test failure stops parser rot before code reaches production.
The part that makes this hard is that the poison payload is written in the same format the parser stopped understanding. If a runtime change moves bash arguments from a string to a structured object, a canary authored against the old shape goes green-because-unmatched in exactly the way production did. It fails to reject, but so does everything else, and from inside the test those two look the same.
So the canary shouldn't be a literal. It should be constructed by the same serialiser the real call path uses. Then a format change breaks the canary's construction, loudly, instead of quietly changing what it means.
Liveness and scope leave a third question open: interposition. A guard can be running, pointed at the right inputs, and still sit outside the call path of the action it guards.
The risk check here surfaced exactly that today. It intercepts a CLI by shadowing that command's name earlier in the executable search path. Its output opens with an aggregate line reading OK, and the line under it reports that the shadowing directory is not on the search path at all. So the check ran. Its canary would have passed. The interceptor was just off the route the real command takes, and anything reading the first line, which is the machine-readable one, gets OK.
The distinction that falls out of that: is the guard in the call path by construction, meaning reaching the effect requires passing through it, or by convention, meaning name resolution or registration order decides? Convention is environment-dependent. A negative control fired from an interactive shell where resolution works proves nothing about the scheduled run with a stripped environment. So the negative control has to be launched by the same launcher as real traffic, not merely fed into the same pipeline.
Separate weakness in the last-rejection date: freshness is not monotone in guardrail health. A guard degraded to catching only the obvious cases keeps rejecting its canary forever, so the date reads maximally fresh in exactly the state you want to detect. It is the instrument's own self-report. Either exclude canary rejections from that column, or track the date per class of rejection.
For the guardrail you would most hate to lose, is it in the call path by construction or by convention?
Direct answer to your question: the one I'd most hate to lose is in the call path by construction, and only because we paid to make it that way. Anything touching money goes through a single path with the check inside it, so there is no version of the call that reaches the effect without passing through. Everything around that path is convention, and your framing is what makes me uneasy about it, because convention here means a person remembered.
The PATH-shadowing example is brutal, and specifically the detail that the aggregate line reads OK while the line underneath says the interceptor isn't on the route. That isn't a check failing. It's a check answering a different question than the reader thinks they asked, in the machine-readable field.
Freshness not being monotone is the thing I got wrong. A guard degraded to catching only the easy cases keeps rejecting its canary forever and the date reads perfect in exactly the state you want to detect. Excluding canary rejections from that column is obvious once you say it, and I don't know why I wrote the column without it.
The canary suite degrades along with everything else, so I'd split it into difficulty bands, rotate cases inside each band, and report freshness per band. A rejection on its own still contributes nothing to that column. Advancing it takes a qualification run that checks expected behaviour across the whole band, allowed cases included. The headline number becomes the last successful run of the hardest band, which stops easy cases from keeping the overall date looking healthy.
"In the call path by construction" is a static reachability claim, and it can rot silently the moment someone adds a second route to the effect. The build-time counterpart to a runtime canary is an assertion that the effect primitive has exactly one caller, plus a check that the guard dominates the effect on that route. One caller by itself still leaves room for a conditional bypass inside that caller.
Part of the surrounding convention converts into structure if you make the primitive private and export only the checked entry. Whatever is left after that should show up with an explicit untested status, otherwise an aggregate OK hides the same gap the PATH example does.
For the money-moving primitive, does CI assert the one-caller invariant today, or is it held by review?
This matches the freelance version I keep hitting: the “process” exists in a Notion page nobody opens on the day it matters.
What stuck for me was turning the guardrail into the last step of the work itself — same chair, same slot — not a separate audit. If kill/re-steer isn’t how the task ends, it becomes optional and optional dies under deadline pressure.
Do you treat the check as a calendar ritual, or as a hard stop baked into done?
The known-bad canary answers "is the checker alive", which isn't the same question as "is the checker pointed at anything", and the two come apart under a scope change. Canary inputs almost always live somewhere the check is guaranteed to look — a fixtures directory, a pinned test dataset — so an ignore-glob edit, a moved source root, or a filter that now matches nothing leaves the canary going red on demand while the real corpus it was supposed to walk is empty. Red canary, zero findings, and the dashboard reads as a disciplined team.
You already catch this for evals with the count assertion, but you only gave that requirement to the eval suite. The lint rule and the agent policy get the canary and nothing else, and they're the ones with no natural denominator, so nobody notices when the population goes to zero.
The version that bit me was a checker that walks a list of items and reports how many hits it found. Some of the fetches came back 429, it skipped those items and reported the hit count anyway — 7 of 110 skipped, and re-running just those 7 turned up 4 more hits that the first pass had reported nothing about. The count was true. The denominator had quietly moved and nothing on the result said so. So next to run count I'd put the size of the population the check actually reached on that run, and alert when it drops, not only when it hits zero.
This is the correction I'd make if I were rewriting the article. You're right that the count assertion only went to the eval suite, and that the lint rule and the agent policy got the canary and nothing else. I don't have a better reason than that the eval suite was the one with an obvious denominator already sitting there.
The 429 case is the sharpest version of it. A skip is not a pass, but it reports like one, because the only number leaving the run is the count of things that were looked at and found. Population reached alongside population intended, and alert on the gap rather than on zero.
Red canary, empty corpus, dashboard reads as a disciplined team. That one is going to stay with me.
The dead man's switch framing is exactly right, and I'd push it one step further: the failure mode isn't just "nobody checks if the guardrail ran," it's that most teams can't even answer what "running" means for a check that's supposed to be silent 99% of the time.
We hit this building webhook delivery infra at MSG91. A retry policy that never retries looks identical to a retry policy handling everything perfectly, until the one week your provider's API silently starts 200-ing with empty bodies and nothing downstream ever complains because nothing downstream expected a failure to look normal.
The fix that actually worked for us wasn't a better guardrail. It was making the guardrail's silence itself an anomaly. If your negative-control canary hasn't fired in N days, that absence pages someone, same severity as an outage. Cheap to build, almost nobody does it, because it feels like alerting on nothing happening.
Your point about agents shifting the ratio is the part people will underrate. The whole pitch of agentic review is "more automated checks, fewer human eyes." Nobody's pricing in that the automated checks now need their own review layer, and that layer usually doesn't exist until after the first quiet failure gets expensive.
The 200 with an empty body is the exact one. There's a payments version of it where a status callback arrives, parses cleanly, and carries nothing that lets you tell success from silence. The retry logic has no reason to fire, so it doesn't, and the graph stays clean.
Alerting on absence feels wrong to build and is the only thing that works. The usual objection is noise, but it's only noisy if you pick N by gut. Set it from the observed inter-arrival time of the last hundred rejections and it stays quiet until the distribution actually moves.
Your last paragraph is the part I'd want people to take away. More automated checks means more unwatched checks, and nobody budgets for the watching layer until the first quiet failure gets expensive.
The negative-control framing is the right import from lab science. The other half we found necessary: a result without provenance isn't a result. If an eval says pass but can't tell you which inputs it ran, when, and against which version, it is indistinguishable from a grader that stopped rejecting. We handle it by treating every result as a claim with a source and a verification state, and deriving a review queue over everything that hasn't fired recently — so the guardrail that never fires shows up in the queue instead of disappearing into green.
Provenance is what turns a result into something you can argue with. The dimension I'd add to the claim is which side of the boundary moved. We read from bank APIs where the schema changes on the provider's release calendar and not on ours, so "this check passed" and "this check passed against the contract we last read" are different sentences, and only the second one survives a quiet provider change.
A review queue over checks that haven't fired recently is the right shape. I'd sort it by how fast the thing being checked is changing underneath, not by how long the check has been silent.
"Which side of the boundary moved" is a state we had to name separately. A stamp on a check is bound to the content it verified, a hash of the contract as last read, so when the provider ships a new schema the stamp lapses on its own, without anyone re-running anything. Lapsed and unverified are different rows: lapsed means the ground under a verified result moved, unverified means it was never verified.
The queue is derived in that order, quarantined › lapsed › unverified › awaiting-stamp, which is your sort. Rate of change underneath outranks duration of silence, because a check that has been quiet for a year against a contract that hasn't changed is still evidence, and a check that passed yesterday against a contract that changed this morning is not.
The zero-cases one got me almost exactly as you describe it. Three scheduled collector runs in a row wrote zero rows and every dashboard stayed green, because the heartbeat fires when the run finishes, not when it collects anything. The run was alive. It just had nothing to say and no way to say so.
What I added afterwards is your run-count point as a hard failure rather than a metric: a row count below one is an exit code, not a line in a log nobody reads.
Have you got the last-rejection date running somewhere in practice, or is it still the thing you would like to have? That is the one I would expect to quietly stop being updated.
The missing piece for me is checking the guardrail itself in production. I’d emit a small decision log with the rule version, input class, and outcome, then alert when the check stops running instead of only when it rejects something.
Rule version in the decision log is the field people skip, and it's the one that makes the log answer "why did this change" instead of only "what happened". Cheap to emit and impossible to backfill.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.