Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
For two field tests, one number refused to move: golden recall sat at 0.087. The reports both named the same culprit — "replay and matcher calibration is now the highest-value engineering target." So we did the obvious thing and worked on the matcher.
The matcher was not the problem. The denominator was.
The symptom: a good rule scored almost zero
Replay grades a candidate trigger against reference trajectories. Recall is the fraction of failures the trigger catches:
precision = prevented / (prevented + broken)
recall = prevented / total_failures
A v0.2.0 candidate that prevented 3 real failures — a genuinely useful rule — was scored 3 / ~200 = 0.015. No threshold can admit 0.015 without also admitting noise. The rule wasn't weak. The denominator was the entire corpus.
Ask the question the metric was actually answering: "does this git rule also prevent docker, python, terraform, and browser failures?" Of course it doesn't. It was never supposed to.
The assumption we made (and where it broke)
We built replay on one implicit assumption:
Assumption: every failure in the reference pool is a fair test case for every candidate rule.
That assumption is wrong whenever candidates are domain-specific — which, for a real rule engine, is always. The consequence is subtle and dangerous: the metric doesn't fail loudly. It just makes good rules look worthless, uniformly, across every model. And a metric that makes everything look equally bad is a metric that hides where the actual problem is.
We also assumed recall was a model measurement. It is a ratio, and we were controlling the denominator as if it were irrelevant.
The fix: scope the reference pool to the source domain (§708)
v0.3.0 filters reference_trajs to the source trajectory's domain before replay:
# scope the comparison set to the candidate's own domain
references = [t for t in reference_pool if t.domain == candidate.source_domain]
prevented = sum(1 for t in references if rule_matches(candidate, t) and not t.success)
recall = prevented / len([t for t in references if not t.success])
The pool did not shrink — it grew from 230 to 444 references. What changed is the slice each candidate is judged against:
| Candidate domain | Scoped references | Global pool (old) |
|---|---|---|
| git | 19 | ~200 |
| python | 30 | ~200 |
| docker | 30 | ~200 |
A rule that prevents 3 git failures is now scored 3 / 19 ≈ 0.16, not 3 / 200 = 0.015.
We made one more change alongside it, and it deserves to be stated plainly: the pass threshold dropped from 0.8 to 0.5. That is not a matcher fix. It is the acknowledgement that once scores are computed honestly, 0.8 was calibrated to an inflated scale. Report both or you are hiding the trick.
The data
Golden and failures/positive recall, both cloud models, pre- and post-fix:
| Corpus | Model | Recall pre | Recall post | Delta |
|---|---|---|---|---|
| golden | gpt-4o-mini | 0.068 | 0.170 | 2.5× |
| golden | llama-3.1-8b | 0.104 | 0.228 | 2.2× |
| failures/positive | gpt-4o-mini | 0.087¹ | 0.182 | ~2.1× |
| failures/positive | llama-3.1-8b | 0.087¹ | 0.259 | ~3.0× |
¹ Golden pre/post are the v0.3.0 pre/post domain-scoping runs (report §8). The report does not tabulate a separate failures/positive pre-run, so that pre column uses the v0.2.0 baseline (0.087); the post values are recomputed from the committed v0.3.0 artifacts (mean best-candidate recall).
No model change. No prompt change. No change to the core matcher scoring. Golden pass rate moved to 40% (gpt-4o-mini) and 50% (llama-3.1-8b) after the near-miss tolerance band. Verified caveat: the committed per-candidate verdicts show **30% for both* models (3/10); the 40–50% is the tolerance-band-applied value and the band-applied re-run was not committed to field-test/results/0.3.0 — see the field-test report's data-verification note.*
What worked
- Recall became diagnosable. Against a global pool, everything looked equally hopeless. Against a scoped pool, the recall signal separates a specific rule from a vague one.
- The fix is model-independent. It held across both models — the signature of an evaluation bug, not a capability ceiling.
- It exposed the real bottleneck. Once the denominator was honest, the remaining failures pointed at the matcher's inability to read paraphrases — a concrete, addressable problem.
What didn't work
- Recall is still far below target. 0.170–0.228 against a 0.70-ish ambition. Doubling a small number leaves a small number.
- The paraphrase gap is untouched. Semantic matching runs at a 0.2 blend weight and cannot bridge "non-fast-forward" against "Updates were rejected because the remote contains work that you do not have locally." Same event, different tokens. The mechanism is worse than the weight suggests: the semantic term is capped at 0.2 of the blend and only floors the score to 0.70 at cosine ≥ 0.80, so a true paraphrase with zero shared tokens scores ~0.2 — below every threshold. Raising the weight alone would not fix it; the floor has to move, and the cosine has to be computed against the failure signature, not the whole trajectory.
- Scoping could flatter reference-rich domains. python and docker have ~30 references; git has 19. We have not measured whether thin domains are systematically penalized.
- Changing two things at once blurs attribution. Scoping and the threshold drop landed together; the 2–3× is scoping's, but the pass-rate movement is both.
Questions we still can't answer
- Would a 0.4–0.5 semantic weight close golden, or does the token-F1 term still dominate the blend?
- Is domain the right scope, or should it be failure-class? We chose domain because the label already exists on every trajectory — a convenience assumption.
- At what reference count per domain does scoping stop helping? We have no curve, only two field tests.
- If a candidate has no domain label, do we fall back to the global pool, and does that reintroduce the bug?
What I learned
Evaluation-set composition is a product decision, not a detail. "Recall against all failures" sounds rigorous and is actually a different question than the one you care about.
Uniformly bad numbers across every model usually mean an evaluation bug. When gpt-4o-mini and llama-3.1-8b agree that everything is hopeless, suspect the denominator before the model.
Scoping is not cherry-picking — if it is principled and pre-registered. We scope by a trajectory property fixed before scoring. We are not removing failures a candidate missed; we are removing failures it was never meant to catch. If that boundary moves after you see the results, you are just tuning the metric until it flatters you.
Recall is a floor. Every reference you add lowers recall without changing the rule. Report it as "at least this many," never as "this many."
The broader lesson
If you score a specific output against a set of examples, ask what the denominator means before you optimize the scorer.
CauterRule spent two releases blaming a matcher for a number the reference pool was manufacturing. The fix was not a better algorithm. It was recognizing that "recall" should mean recall within the problem this rule is about. Change the comparison slice and the same model — the same everything — looks 2–3× better and, more importantly, becomes diagnosable.
References
- CauterRule v0.3.0 release notes
- v0.3.0 field test report
- Per-model cloud comparison
- Threshold calibration (§708 / threshold changes)
- PRD — success metrics
- Corpus format spec (domain labels)
- User guide · Changelog
CauterRule v0.3.0 is released. The full recall data and the 40-corpus × 2-model matrix are in the field test report. The repo is public. Install with
pip install cauterule. Changelog · Release notes
Top comments (11)
Removing the failures a git rule was never meant to catch also removes the docker and python successes it was never meant to touch, and those are the runs the precision half of the grade exists to count. The filter runs before replay, so both halves now come from the same slice. Recall got the denominator it should have had. Precision lost sight of the overgeneralization the sidecar is supposed to separate out, since a git trigger that fires on a docker success can no longer land in broken.
The two terms are asking different questions of the pool. Recall asks whether the rule catches what it was written for, so its pool belongs to the source domain. Precision asks what the rule breaks wherever it is allowed to fire, so its pool follows the runtime, not the trajectory the rule came from. That also moves the unlabeled-candidate question out of evaluation and into deployment. If a promoted rule only fires on trajectories carrying its own label, the scoped pool is honest for both terms, and a missing label has to mean silence rather than a fallback. If it fires wherever the trigger matches, the global pool was only ever the bug for recall.
Thanks Edward — this is a sharp catch, and you're right. Scoping the reference pool to the source domain (#708) fixed recall's denominator but quietly scoped precision's broken set to the same slice, so a git rule firing on a docker/python success can no longer land in broken. The filter runs before replay (scripts/run-field-test.py), and build_evidence_report derives precision and recall from that one list — so both halves now answer the same question when they should be asking different ones. Exactly your point: recall's pool is the source domain, precision's pool is wherever the trigger is allowed to fire.
Your second observation is the one I find most useful: it moves the unlabeled-candidate question out of evaluation and into deployment. If a promoted rule only fires on trajectories carrying its own label, the scoped pool is honest for both terms and a missing label must mean silence rather than a fallback. If it fires wherever the trigger matches, the global pool was only ever the bug for recall.
I've filed this as a new issue with the full analysis, evidence table, and the code paths involved (plus the #723 domain gate, which compounds it):
github.com/deghosal-2026/CauterRul...
It's tracked as a methodology fix distinct from the same-domain over-fire in #815, and cross-linked to #708, #723, #720, #814 and #578. Thanks for reading the methodology closely enough to find the seam — this is the kind of feedback that actually moves the metric design.
The denominator mistake is sneaky because nothing errors. I grade my own moderation heuristics the same lazy way at first: a rule that caught three real spam runs, scored against every message the board has ever seen, ends up looking like it does nothing. Scoping the pool to the failure class the rule was written for moved the numbers by an order of magnitude without touching the matcher. The dangerous part is the one you named: a metric that makes everything look equally bad sends you debugging the wrong component with total confidence.
the 0.087 baseline is a useful number because it makes the domain scoped replay result readable: if recall had started at 0.6, doubling it would have just looked like noise reduction. you needed the floor low enough that the gain was obvious.
what i'd want to see is whether the standing rules from v0.3.0 degrade over time as the corpus drifts, or whether they stay stable. a rule that extracts from a failure trajectory is anchored to one distribution — i'd expect it to help a lot immediately and then need a reseed on a new domain.
does CauterRule track rule age or hit rate over time?
the denominator framing is the real insight. ‘does this git rule also prevent docker, python, terraform, and browser failures?’ is exactly the question you stop asking when you’re staring at 0.087 and blaming the model.
we hit this building evals for document classifiers. recall looked catastrophic across the full corpus, scoped it to document type and it tripled. the metric wasn’t wrong, we were asking it the wrong question.
the silent failure mode you named costs the most: a metric that makes everything look equally bad hides where the problem actually is.
how does v0.3.0 handle domain assignment when a trajectory spans multiple domains?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.