DEV Community

Debashish Ghosal
Debashish Ghosal

Posted on AI-assisted

The Soak That Reported 5,003 Deliveries for 5,000 Records

#ai #testing #programming

5,003 deliveries. 5,000 records. Three extra.

In a streaming system whose entire pitch is the store is the truth, that's the scariest output a test can produce. Three duplicate deliveries means somewhere, something sent twice. It means exactly-once is once-short. My first annotation in the field test log was: real defect — duplicate deliveries.

The root cause turned out to be three records I put there myself. And the part I want to keep from this isn't the bug — it's that the gate flagged a genuine inconsistency, and the inconsistency was in my test's fence, not the product's.

This is from the agentwatch v0.2.1 release — the agent-security tool in agentic-suite. Full forensics in the v0.2.0 field test report.

What the soak does

The case is FT-STR-2, the drop-consumer reconciliation. The scenario it models is the one that actually keeps streaming systems honest: the consumer — the thing watching the record stream — dies mid-stream. Crashes, loses its place, comes back. What did it miss?

The test streams 5,000 events through the live tail, kills the consumer partway, lets the store keep ingesting, restarts, and then reconciles: what the consumer was told it received versus what the store actually holds. If those diverge, the operator's view is lying about the record. That's the whole trust proposition.

The original assertion on this case was comfortable:

delivered >= records   # "no store loss" — the consumer saw everything
Enter fullscreen mode Exit fullscreen mode

Fine as far as it goes. Also — as the hardening pass immediately demonstrated — not the actual contract.

The stricter gate was already in the product

The product has its own reconciliation predicate, within_bounds. Not delivered >= records. delivered == records. Exactly. Because for this store, "I delivered more than I hold records for" isn't a success condition — it's an accounting error. Deliveries that exceed records mean something is being counted that was never recorded, which is a different way of saying the same lie the store exists to prevent.

So the hardening pass did the simplest, most correct thing: throw out my lenient reading and assert the product's own gate.

It went red on the first run. delivered=5003, records=5000, within_bounds=False.

Three records, ambient

The 5,000 test events were all accounted for. The three extras were the daemon's attestation records — the recorder periodically writes "I am the recorder, here's my state" entries into the store it owns. Three of them landed in the store during the soak window.

And the soak wasn't soaking against a store the test owned. It was pointed at the recorder's live store — the real one, with the real daemon writing into it. The delivery count swept up everything the store knew about, which was 5,000 test events plus 3 attestations.

Same class of failure, three times in one hardening week:

Case Looked like Was
FT-STR-2 duplicate deliveries (product bug) soak ran against a shared, ambient store
compliance report nothing written (product bug) --out writes a file; my driver globbed it as a directory
OTel join zero joins (product bug) the fixture is OTel-only — no hook records exist to join

Two of three, the harness. One pattern: the test environment asserting about state it didn't exclusively own.

The fix, and the retraction

The soak now runs against a fresh temp store it creates and owns — no daemon writing into it, no ambient records, nothing in it that the test didn't put there:

delivered=5000, records=5000, within_bounds=True
Enter fullscreen mode Exit fullscreen mode

Deterministic. Re-run, green — for a reason this time.

And the field test log did something I now consider non-negotiable: it recorded the retraction explicitly. The running-fixes log shows FT-STR-2 first as real defect — duplicate deliveries, then reclassified: harness artifact, not a product defect. Both entries stayed in the report. A defect classification is a hypothesis; root cause is allowed to move it; the journal keeps both so the next reader knows how the reasoning went.

tip: if your test asserts a count against shared state, you haven't written a test — you've written a race with extra steps. Count only against state you created.

Why I keep saying the gate was right

The tempting narrative is "false alarm, waste of time." That's the wrong read, and I want to be precise about why.

within_bounds did exactly its job: it flagged a real inconsistency between deliveries and records. The inconsistency was in the test setup — but it was real, it was material, and under the lenient >= reading it had been passing silently the whole time. The strict gate didn't cry wolf. The strict gate found the first wolf available, and the wolf happened to be standing in my own doorway.

If the ambient records had been fewer records instead of more — if the daemon had rewritten or compacted the store mid-soak — the same fence would have caught that too. The gate doesn't know which side the error is on. That's what makes it a gate rather than a cheerleader.

There's a companion piece in the first season of this series — a soak that found a daemon dropping records behind an O(n) lock. Same instrument, opposite direction: that one caught a real product defect, this one caught a test defect. Two soaks, two verdicts, same lesson both times — the soak doesn't tell you where the problem is. It tells you the books don't balance. The classification work after that is yours.

What I keep from this

  1. Soak against a store you own. Shared state isn't realism, it's ambient noise with a deadline attached. A fresh temp store made the test deterministic and meaningful in one move.
  2. Assert the product's own correctness gate, not a laxer reading of it. >= had been passing forever. == failed in ninety seconds. The stricter predicate wasn't harder to write — it was just the one the product actually promises.
  3. Classify before you fix. Product defect vs. harness artifact vs. test-environment contamination are three different work items, and only one of them touches the product. Two of my three reds that week wanted zero product changes.
  4. Retract loudly. The report keeps the wrong classification and the right one, in order. A field test journal that never admits a reclassification is a journal that's editing for optics.

All of it shipped in agentwatch v0.2.1 — on PyPI.

The question worth taking to your own reconciliation tests: what state does your count run against, and did you create all of it? If you can't answer that in one sentence, your next "impossible" failure mode is probably already scheduled.

References

Top comments (0)