0/60 Wasn't the Model: The Empty Haystack Behind My Two Worst Corpora

Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

Every release, the field test produces a list of worst corpora, and every release the honest first guess is the same: the model can't do this kind of failure. The two worst carried into the v0.3.1 cycle from the v0.3.0 report: adapters — 0/60 on both models, 100% matcher_gap — and raw/ci — 0/110.

Both diagnoses were wrong. Both fixes were data. And both corpora now pass: 60/60 and 21–26/47.

That is the whole article: a 0% corpus is a question, not a verdict, and the question you should ask before touching the model is "what is in the haystack?"

The diagnostic that changed the plan: matcher_gap is not no_signal

Replay attributes every inconclusive to a reason. The two that matter here are different animals:

adapters was 100% matcher_gap. The model was happily extracting LangGraph, CrewAI, and PydanticAI rules. They just had nowhere to land: the reference pool had zero framework-specific trajectories. A candidate with nothing to match against is not evidence the candidate is bad. It is evidence the haystack is empty in that domain.

raw/ci told a messier version of the same story — and the haystack was broken in four layers at once.

Fix 1 (#726): the adapters pool was empty. Fill it.

The reference pool for v0.3.0 had no langgraph/crewai/pydanticai trajectories at all, so every adapter candidate fell back to a pool with no same-domain references and scored matcher_gap. The fix was not a matcher change. It was authoring adapter-specific reference signatures with matching domains — same shape, same domain label, real failure text.

The result, both models, no model change:

0 → 100% in one cycle. The strongest before/after in the sweep. The model had been right all along; the pool had been empty.

Fix 2 (J11): the CI haystack was broken in four layers

raw/ci went from 0/110 to 21/47 (gpt) and 26/47 (llama) — a ~45–55% pass corpus out of a zero corpus. Four separate data bugs, each one independently capable of producing a zero:

1. The collector read the wrong end of the log. collect-ci-corpus-v2.py's predecessor stored the first 2000 characters of gh run view --log — the CI runner's boot/checkout boilerplate. The actual failure is at the bottom of a CI log. So 99 of 110 trajectories had no failure text in them. The model was being asked to extract a rule from a log that did not contain the failure. Re-fetch the tail, re-derive failure_class.
2. The corpus was padded with noise. 34 infra-only + 28 bogus trajectories carried no signal. Deleted: 110 → 48.
3. The reference bucket was generic. 18 generic CI references for a corpus of real, repo-specific CI failures. Authored 48 same-domain CI sibling references (generate_ci_replay_refs.py); the pool grew 540 → 588.
4. The signature was diluted. The failure_class label was mixed into the embedded signature, dragging short paraphrase triggers under the floor (more on the floor itself in the next article). The matcher now takes the max similarity over a class-free failure view.

Layer 1 is the one that generalizes furthest. A log collector that grabs the head of a file is measuring boilerplate and calling it the event. If your "failure text" field is ever empty more often than you expect, check which end of the stream you are slicing before you check the model.

The model gap that remained (and was accepted)

Once the haystack was real, the two models diverged in an instructive way on raw/ci — 21/47 for gpt-4o-mini, 26/47 for llama-3.1-8b, with 11/47 trajectories discordant (8 llama-only passes vs 3 gpt-only). The paired-delta analysis (J15) showed why: gpt writes repo-specific triggers — exact error codes, module names — that score 0.0–0.11 against generic sibling references, while llama's broader phrasings match better.

We did not fix that. We accepted it (J14) and reported it, because the honest root cause is a model/style gap against a reference pool that is still too generic to reward specificity. One confession belongs in this article too: the earlier plan to "lower the raw/ci threshold from 0.45 to 0.40" was based on a dead code path — CORPUS_THRESHOLDS is never read; the live threshold for raw/* is 0.35 via threshold_for_corpus(). A threshold change that moves no number is a threshold you have not traced to the call site.

What worked

• Treating matcher_gap as a first-class signal. It is the pool's name on the problem. A corpus that is 100% matcher_gap should trigger "inventory the reference pool," not "upgrade the model."
• Authoring same-domain references. For both adapters and raw/ci, the fix was adding references the candidate was actually about. The pool is a product surface, not a fixture.
• Repairing the corpus before tuning the model. 62 deleted trajectories, one collector rewrite, 48 new references — then, and only then, the pass rate became a measurement of the model instead of a measurement of the collector.
• Paired per-trajectory deltas (J15) made the gpt/llama split diagnosable instead of a "one model is better" hand-wave.

What didn't work

• The v0.3.0 read of 0/60 as a capability verdict. "The model can't extract framework rules" was written in a report before anyone checked whether the pool contained a single framework trajectory. A zero on an empty haystack is the emptiest of signals.
• The head-of-log collector. It ran for a release and a half, feeding the model boilerplate, and the failure was invisible because the model's "weakness" was exactly where boilerplate produces no rule.
• Threshold reasoning on dead code. The 0.45→0.40 option cost real thinking time before someone traced CORPUS_THRESHOLDS to zero call sites.

Questions we still can't answer

• J18 — the 0-accepted corpora that survived. public/staleness 0/10, public/synthetic 0/10, lifecycle 0/40, mcp 0/20, gpt public/domains 0/30. For each: coverage gap (fix with references) or matcher gap (fix with config) or model gap (accept)? diagnose_corpus.py exists to answer this per corpus; the answers are the v0.4.0 work list.
• How many references per domain before returns flatten? adapters went 0→60/60 with a modest reference addition. golden needed 54 authored replay references. Is there a knee in the curve, or do thin domains (raw/sibling-repos: 1/10 on both models) keep starving?
• Do authored references overfit the test? We author references in the domain of the corpus we then test on. That is how you fix coverage — but it is also how a self-graded loop quietly forms. The public/* corpora are the honest check on whether authored-pool gains transfer, and they are precisely the ones still at zero.

What I learned

A 0% corpus is a question. "What can the grader see?" — and the answer is usually the reference pool, the corpus collector, or both. The model is the last suspect, not the first.

matcher_gap is a pointer to the data layer. When a candidate exists and nothing in the pool resembles it, the pool is on the hook. Read the attribution before you read the rate.

The reference pool is a product surface. It decides what a rule can be compared against, which means it decides what counts as verified. Treating it as a static fixture is how you ship a ruler with holes.

Check which end of the stream you sliced. A log, a trace, a transcript — the event is rarely at the top. Collectors that grab the head produce "weak model" readings that are really "empty input" readings.

The broader lesson

Before you blame the model, inventory what the grader can see. If the answer isn't in the haystack, no model scores — and the 0% you are reading is a property of the evaluation, not the capability.

The haystack is full now. The next problem was a number that sat between the model and the verdict: a similarity floor set at 0.80, in a world where honest paraphrase matches land at 0.63.

References

• CauterRule v0.3.1 release notes
• v0.3.1 field test report (§6 per-corpus results, Appendix A J11/J14/J15/J18)
• CI corpus collector (tail-of-log, re-derived failure_class)
• CI sibling reference generator · golden replay reference generator
• Threshold calibration (live vs dead threshold paths)
• Auto-generated per-corpus results · Changelog

CauterRule v0.3.1 is released. The adapter and CI before/after, the paired model deltas, and the J11/J18 issue journal are in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes