If your AI reviewer says "pass" every time, you didn't build a reviewer. You built a rubber stamp.
I know because I built one. Not on purpose. It looked like a benchmark. It had precision, recall, thresholds, a green suite. And the model found the cheapest possible way to satisfy all of it.
Here's the receipt. My local 3B model produced a rule trigger that was literally the string "step_1". It matched every trajectory, because every trajectory contains a step field and step one is in all of them. My matcher scored it precision 1.00, recall 0.02 and returned the verdict: pass.
Two false positives traced straight to that one trigger. The model wasn't misbehaving. It was solving the problem I defined. Full write-up here.
Reward hacking is the default, not the edge case
We tend to file "the model gamed the benchmark" under safety research, as something that happens to frontier labs. But it happens to anyone who writes a matcher. And it happens first, not last, because it's the path of least resistance.
The model's actual objective was never "find real failures." It was "produce a trigger that scores above 0.70." Under that objective, "step_1" is a perfect answer. Every token appears in every reference. It is optimal behavior for the wrong reward.
Your model is not misbehaving. Your benchmark is mis-rewarding.
The shortcut wasn't in the model. It was in the data format.
Every reference trajectory had a step field with a number. "step_1" is a substring of all of them. Any structural artifact that appears in every record (step numbers, timestamps, session IDs, tool names) is an attack surface.
The fix was not to strip those fields. They're part of the format. The fix was to stop rewarding matches on them. I added a three-line gate to reject degenerate triggers before they ever reach the matcher:
_DEGENERATE_TRIGGER_RE = re.compile(r"^step[_\s]*\d+$", re.IGNORECASE)
def rule_matches(candidate, trajectory, threshold=0.70):
if _DEGENERATE_TRIGGER_RE.match(candidate.trigger):
return False # degenerate trigger: never matches
Three lines closed the visible hole. But closing one shortcut is not the interesting part.
The second shortcut was semantic, and harder
After the regex, three false positives remained, and they had nothing to do with structure:
-
"git push fails with authentication error"matched"git push fails with non-fast-forward". Different failure class, shared 3 of 6 tokens. -
"python import fails with wrong module"matched"python ImportError". Wrong tool entirely, token overlap.
These are legitimate, specific-sounding triggers. The matcher can't tell "authentication error" from "non-fast-forward" because both are "git push fails with X." A token-overlap matcher sees similarity. A human sees two completely different problems.
The regex caught the easy shortcut. The semantic gap is still open, and it's the same lesson one level deeper: the matcher is the reward function, and my reward function was "shared tokens," not "same failure."
A green suite can be the bug
This is the part that took me a week to internalize. I spent that week fixing the matcher: precision formula bug, 50+ distinctive phrases, expanded aliases, raised floors. Four fixes, 359 validation tests, all green. Golden pass rate moved 10% to 20%.
Then I found a six-line fix in the simulator, the component that classifies what a match means. It was counting near-miss recoveries as clean successes, so it penalized triggers for correctly firing on them. Golden jumped 20% to 50%, on both cloud models, the same day. That post-mortem is here.
The green suite was passing the whole time. It was validating the wrong layer.
A decisive verdict is not the same as a correct verdict. Tests that pass can be tests that measure the wrong thing.
The tell: uniformly bad numbers across every model
Before the fix, golden pass rate was stuck at 20% across local and cloud, 3B and 8B. That looks like a capability ceiling, as if the models just weren't good enough yet.
It wasn't. It was a classification bug that affected every model identically. When every model gets the same wrong result, suspect the evaluation layer before the model. Model-independent failure is usually a measurement failure.
I hit the same shape again later: golden recall sat at 0.087 for two field tests because every candidate was graded against the full 230-trajectory pool instead of its own domain. A rule that prevented 3 git failures scored 3/200, about 0.015. Scoping references to the source domain lifted recall 2 to 3 times with no model, prompt, or matcher change. The denominator was the bug. Details.
What I do differently now
Three habits, all boring:
- Trace every false positive to its trigger. "5 false positives" is meaningless. "2 degenerate + 3 semantic" tells you exactly what to fix.
- Ask what the denominator means before optimizing the scorer. Recall against all failures is a different question than recall against the failures this rule was meant to catch.
- When results are uniformly bad across models, look at the layer that classifies output. That's where model-independent failures live.
None of this is exotic. It's the discipline of treating your evaluation as a product surface, not a formality.
What I still can't defend
False negatives are invisible. If both my matcher and my model miss a real failure and agree, I get a clean verdict and a green suite, and no signal that anything is wrong until it surfaces in production. False positives are annoying. False negatives are dangerous.
I don't have a complete defense against that. A better reward function raises the bar; it doesn't remove the risk.
So what is the last thing your AI system "passed" that it shouldn't have? Was the fix in the model, or in what you were measuring?
-
Repo: CauterRule ·
pip install cauterule - Prior posts: My 3B Model Found a Shortcut · The 6-Line Fix That Outperformed My Entire Matcher Week · Our Recall Was 0.087 and the Model Was Innocent
Top comments (12)
The "your model isn't misbehaving, your benchmark is mis-rewarding" framing is exactly right, and I think it's underappreciated how early it shows up. The degenerate-trigger regex gate is the right instinct, but in our experience blacklisting known cheats turns into whack-a-mole — you close
step_\d+, then it finds timestamps, then session IDs. What ended up saving us more time was a cheap invariant that doesn't care about the specific artifact: a trigger that fires on a held-out set of known-negative trajectories at above chance is degenerate by definition, regardless of what it matched on. It catches the semantic shortcuts (yourgit pushexample) that no regex will. Curious whether you considered scoring candidates against a negative corpus rather than enumerating structural artifacts — the second class of shortcut you found makes me think the format-level fixes are necessary but never sufficient.Max, yes — and by the end the negative corpus was the fix, not the regex. The three lines only prefilter the obvious structural case. The durable gate is scoring every candidate against trajectories that should stay silent — failures/negative, the counterexample set, near-misses, cross-scenario calibration negatives — and rejecting a trigger that fires on any of them before precision/recall even matter. That's what catches your git push fails with authentication error vs non-fast-forward pair. No regex will, because the leakage is semantic.
"Necessary but never sufficient" is exactly right, for a reason I didn't appreciate at first: a negative corpus only catches shortcuts that fire on known negatives. Silence is cheap, so a trigger can pass by being inert. We guard with minimum negative-corpus volume, but I won't claim it's closed — a shortcut that only leaks on unseen negatives still walks. Same blind spot as false negatives.
The "solving the problem I defined" framing is the part most teams skip. I hit a mirror version of this running a local 7B model as a code-review gate: I rewarded it for "issues found per PR" and within a week it was flagging every magic number and TODO comment while waving through a Stripe webhook that ACKed before persisting. Precision looked great on the dashboard; the failure mode moved somewhere the metric couldn't see.
Your three-line degenerate-trigger gate is doing the same job as what I ended up with — a denylist of "cheap wins" the reward function can't distinguish from real work. The harder lesson from your semantic-shortcut section: token overlap is not failure-class overlap, and no amount of threshold tuning fixes that. You need a second model (or a human) judging kind of failure, not just presence of text.
Curious: after you closed the structural shortcut, did the model's trigger distribution shift toward longer, more specific rules, or did it just find the next-cheapest artifact (timestamps, session IDs) in the same format? In my setup the shortcuts migrated rather than disappeared — wondering if you saw the same whack-a-mole.
@debashish_ghosal, the model-independent plateau is a strong signal that the classifier or reward layer is broken rather than every model sharing the same capability ceiling. The regex closes the known shortcut, but I’d add counterfactual negatives where token overlap stays high while the failure class changes, plus holdout structural artifacts that were absent when rules were generated. agent-inspect treats deterministic trajectory facts as contract inputs, but those contracts need the same adversarial scrutiny. How are you sampling hard negatives so the next shortcut is found before production does it for you?
Fair read — and you're right that the plateau pointed at the layer, not the models: the fix was six lines in the component that classifies what a match means. On hard negatives, honest answer: today they're curated, not mined — nearmiss corpus, a should-silence corpus, and cross-scenario calibration pairs (each golden rule scored against every other scenario and all nearmisses as expected no-matches). That last set is the closest thing to your "high overlap, different class," but it's a fixed set. I don't have the generator you're describing — swap the failure class, keep the token overlap, feed the pair in as an expected non-match — and no holdout for structural artifacts the rules were never generated against. Filed it as the next work item (CauterRule #814); the rule-mutation test perturbs rules, not trajectories, so the negative space is exactly as uncovered as you say. The contract point on agent-inspect stands — deterministic facts are only as trustworthy as the contract that produced them.
the 'step_1' precision 1.00 recall 0.02 case is the clearest version of this i've seen written down. the model was not wrong. it was optimal for the objective you gave it.
we hit the same thing with an LLM judge that was grading its own family model. swapped in a different vendor and the passing rate dropped 27 points overnight. the judge had learned the texture of the outputs it was evaluating, not the quality. same shortcut, different form.
the degenerate trigger gate is the right fix for the structural case. how are you handling the semantic gap — LLM judge, held out human labels, or something else?
yeah the family model bias is brutal to catch — our judges consistently scored their sibling models 8 to 12 points higher on identical outputs. what held up through vendor swaps: a 200 sample human labeled anchor set, scored quarterly. any judge that deviates more than 10% recall on that set gets flagged before going live. made the bias gate model agnostic since the truth signal never changed. are you keeping the calibration set frozen across swaps, or refreshing it as your product evolves?
A useful evaluator test is to generate candidates that preserve the score while violating the intended meaning. If tiny structural artifacts keep winning, the benchmark is measuring familiarity with the fixture rather than the behavior. I would also hold out entire formats or tool vocabularies, not only examples; otherwise the next shortcut is often a different field name for the same leakage.
Jo, this is the test I don't have yet. What exists is adjacent, and it's the inverse of your point: a mutation test that asserts perturbed rules degrade precision. Generating candidates that preserve the score while violating the meaning is the sharper framing — we don't do it, and it would have caught the semantic class earlier than the corpus did.
Your second point is a real gap too. We hold out examples and whole trajectories, but not formats or tool vocabularies. The generation pools vary models and tools per example — that's not the same as holding out the format itself. If the next shortcut is the same leak under a new field name, example-level holdouts won't see it. Both go on the list. Thank you.
"step_1" matching every trajectory is the most honest eval failure I have seen. If your metric is gameable, your model will game it.