The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split

Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

For two releases we treated "the pass rate is low" as one problem and reached for the extractor. It turned out to be two problems we had fused into a single number. This is the split, and it changed where we point next.

The assumption: one metric, two questions

The pipeline has two halves that answer two different questions:

Extraction: given a failure, does the model produce the right rule?
Replay / evaluation: given a rule, can we verify it against history?

We measured only the second, and we read its score as a verdict on the first. The replay gate that decides promotion computes:

precision = prevented / (prevented + broken)
recall    = prevented / total_failures

where prevented and broken come from simulate(), which calls rule_matches() — a text matcher. So prevented means "the trigger's prose reached a similarity threshold against a reference failure's prose," and broken means "…against a reference success's prose." Nothing in that path asks whether applying the rule's directive would have changed the trajectory's outcome. The gate is a lexical resemblance check wearing a validation costume.

The F-001 example: the rule was right, the grader said no

In the failures/positive corpus, trajectory F-001 is the canonical git case:

The model reproduced the reference rule almost verbatim. Extraction did its job. Then replay scored it:

failures_prevented = 5   successes_broken = 3   near_misses = 1
precision = 0.625        recall = 0.625         verdict = INCONCLUSIVE

Three "broken" successes were S-022-git-pull, S-023-git-status, and NM-044-git-commit-hook — a successful git status counted as broken by a git-push rule, because the two share the token git. A correct rule was demoted to inconclusive by three unrelated successes it merely rhymes with. The model wrote the right rule. The grader rejected it on the prose.

The assumption we made (and where it broke)

Assumption: the replay verdict tells us whether the extracted rule is good.

It tells us whether the extracted trigger's surface form resembles stored reference surface forms. Those are different claims. A correct rule phrased differently scores low; a wrong rule that happens to share vocabulary scores high. The metric is a text-similarity proxy, and it is the gate.

The telling part: the undercount is not unique to replay. The corpus carries a ground-truth rule (expected_rule) on 23/50 failures/positive trajectories and 288 reference-expansion trajectories — a ready-made extraction-accuracy instrument we were never scoring against. When we do score it naively (token-F1 of the extracted rule vs expected_rule), we get only ~0.58 (llama-3.1-8b) / ~0.50 (gpt-4o-mini). Not because the model is wrong — F-001 is near-verbatim — but because it rewords, and a token comparator can't see through rewording. The same paraphrase gap that breaks replay also undercounts extraction. We had one brittle text-matching lens in two places.

The data

Cloud models, failures/positive (23 trajectories carry expected_rule):

The two left-column numbers look bad and point at the model. The right two columns say the model was fine and the measurements are the weak link. The spread is the whole story: extraction is substantively right but lexically variable, and every metric we ship is lexical.

What worked

• Naming the two halves separately stopped the misattribution. "Low pass rate" is a symptom; "extraction accuracy" and "replay fidelity" are the two variables. Once separated, the model stops taking the blame for the grader.
• The ground truth was already in the corpus. expected_rule needs no new labeling on failures/positive — it was being parsed out and dropped. Picking it up is the cheapest high-signal metric we have.
• One worked example did more than any average. F-001 — right rule, inconclusive verdict, three token-sharing successes — is the argument in one line. Averages hid it; the single case exposed the mechanism.

What didn't work

• We had a ground-truth rule and never scored against it. expected_rule is dropped at parse (Trajectory has no such field), so the only quality number was the replay verdict. We graded the homework with the wrong rubric for a full release.
• The replay metric masquerades as validation. "Tested against history before promotion" is the product's core promise, but the test is lexical resemblance. A reader who believes the promise and inspects the gate finds prose matching.
• Token-F1 undercounts extraction too. Reaching only ~0.5–0.6 against known-good rules, a token-F1 extraction score would also wrongly suggest the model is mediocre. The comparator needs a semantic or signature signal, or it repeats the replay's error on the extraction side.
• Both metrics share one brittle primitive. Because replay and the naive extraction check both lean on the same token matcher, fixing one without the other leaves the split half-measured.

Questions we still can't answer

• Should replay mean "does this rule prevent this class of failure" (behavioral) or "does it match this history" (lexical)? We have been shipping the second and calling it the first.
• Can we validate by applying the directive to the reference trajectory and checking the outcome flips — turning prevented from "text matched" into "outcome changed"?
• Is expected_rule dense enough to backfill onto golden (currently null) so extraction quality has a clean per-corpus number everywhere?
• At what point does a rule that is right-but-reworded deserve credit, and who decides — a threshold, an embedding floor, or a human?

What I learned

A pass rate is a composite; decompose it before you optimize it. "8–10% pass" fused extraction and evaluation. The moment we split the number, the model exonerated itself and the grader took the hit.

Ground truth you don't score against is a decoration. The corpus already knew the answer (expected_rule). Not wiring it into a metric meant we optimized the wrong half while the right half went unmeasured.

If your "validation" only reads words, it can't validate meaning. The replay gate grades whether the trigger's prose rhymes with stored prose. That is a retrieval property, not a correctness property — and it is the gate.

One crisp failure case beats a dozen averages. F-001 is the entire diagnosis. When a metric makes a known-good rule fail, the metric is the bug, full stop.

The broader lesson

If you build a loop that produces an artifact and then scores it, keep the two measurements separate and keep them honest. A score that grades surface form instead of behavior will make good outputs look bad and bad outputs that share vocabulary look fine — in both halves of the loop at once.

CauterRule spent two releases tuning an extractor that was already writing the right rule, because the number it was chasing was really a prose-similarity score. The fix is not a better model. It is (a) scoring extraction against the expected_rule we already have, with a comparator that can see past rewording, and (b) making the replay gate check whether the directive changes the outcome, not whether the trigger rhymes. Same model. Same corpus. Two honest metrics instead of one flattering lie.

References

• CauterRule v0.3.0 release notes
• v0.3.0 field test report
• Corpus format spec (expected_rule, domain labels)
• PRD — success metrics
• PRD — architecture
• User guide · Changelog

CauterRule v0.3.0 is released. The replay and extraction data are in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes