Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, rep...
For further actions, you may consider blocking this person and/or reporting abuse
Removing the failures a git rule was never meant to catch also removes the docker and python successes it was never meant to touch, and those are the runs the precision half of the grade exists to count. The filter runs before replay, so both halves now come from the same slice. Recall got the denominator it should have had. Precision lost sight of the overgeneralization the sidecar is supposed to separate out, since a git trigger that fires on a docker success can no longer land in broken.
The two terms are asking different questions of the pool. Recall asks whether the rule catches what it was written for, so its pool belongs to the source domain. Precision asks what the rule breaks wherever it is allowed to fire, so its pool follows the runtime, not the trajectory the rule came from. That also moves the unlabeled-candidate question out of evaluation and into deployment. If a promoted rule only fires on trajectories carrying its own label, the scoped pool is honest for both terms, and a missing label has to mean silence rather than a fallback. If it fires wherever the trigger matches, the global pool was only ever the bug for recall.
Thanks Edward — this is a sharp catch, and you're right. Scoping the reference pool to the source domain (#708) fixed recall's denominator but quietly scoped precision's broken set to the same slice, so a git rule firing on a docker/python success can no longer land in broken. The filter runs before replay (scripts/run-field-test.py), and build_evidence_report derives precision and recall from that one list — so both halves now answer the same question when they should be asking different ones. Exactly your point: recall's pool is the source domain, precision's pool is wherever the trigger is allowed to fire.
Your second observation is the one I find most useful: it moves the unlabeled-candidate question out of evaluation and into deployment. If a promoted rule only fires on trajectories carrying its own label, the scoped pool is honest for both terms and a missing label must mean silence rather than a fallback. If it fires wherever the trigger matches, the global pool was only ever the bug for recall.
I've filed this as a new issue with the full analysis, evidence table, and the code paths involved (plus the #723 domain gate, which compounds it):
github.com/deghosal-2026/CauterRul...
It's tracked as a methodology fix distinct from the same-domain over-fire in #815, and cross-linked to #708, #723, #720, #814 and #578. Thanks for reading the methodology closely enough to find the seam — this is the kind of feedback that actually moves the metric design.
The denominator mistake is sneaky because nothing errors. I grade my own moderation heuristics the same lazy way at first: a rule that caught three real spam runs, scored against every message the board has ever seen, ends up looking like it does nothing. Scoping the pool to the failure class the rule was written for moved the numbers by an order of magnitude without touching the matcher. The dangerous part is the one you named: a metric that makes everything look equally bad sends you debugging the wrong component with total confidence.
the 0.087 baseline is a useful number because it makes the domain scoped replay result readable: if recall had started at 0.6, doubling it would have just looked like noise reduction. you needed the floor low enough that the gain was obvious.
what i'd want to see is whether the standing rules from v0.3.0 degrade over time as the corpus drifts, or whether they stay stable. a rule that extracts from a failure trajectory is anchored to one distribution — i'd expect it to help a lot immediately and then need a reseed on a new domain.
does CauterRule track rule age or hit rate over time?
the denominator framing is the real insight. ‘does this git rule also prevent docker, python, terraform, and browser failures?’ is exactly the question you stop asking when you’re staring at 0.087 and blaming the model.
we hit this building evals for document classifiers. recall looked catastrophic across the full corpus, scoped it to document type and it tripled. the metric wasn’t wrong, we were asking it the wrong question.
the silent failure mode you named costs the most: a metric that makes everything look equally bad hides where the problem actually is.
how does v0.3.0 handle domain assignment when a trajectory spans multiple domains?