I gave a plain Claude Code session a task on a small invoicing API: add credit notes, numbered CN-0001, CN-0002, in sequence. It came back with 42 tests passing and a clean typecheck. Then a hidden finance check failed: a rejected request had burned a number. CN-0004 where CN-0002 was due.
The handler took the next number from the counter before it validated the request. It had copied that order from the existing issue route. Nobody told it not to, because nothing in the repo said so.
Every developer's reflex at that point is a louder line in CLAUDE.md. I wanted to see what Matt Pocock's /retro skill would propose instead, and whether the same task would ship clean afterwards. Can one retro turn a failed session into something the next session cannot ignore?
What I actually tested
The repo is ledgerly, a Node 20 and TypeScript invoicing API with vitest: 18 source files, 34 tests, README conventions, no CLAUDE.md, no CI, no pre-commit hook. I copied it out of my own projects so none of my global instruction files would load.
Pocock's skills came from the official marketplace, installed at project scope: version 1.3.1, which is where /retro lives. The skill reads the session log, ranks what would have prevented the trouble, and edits nothing. Its front matter even blocks the model from invoking it on its own.
Success for me was simple: run the task, run the retro, approve what survives, re-run the task in a fresh session, and watch the finance check pass five out of five.
The retro cannot see a failure the session never saw
The first /retro opened with 'This session was smooth'. Five ranked candidates followed: no guardrail runs the checks, a migration rule that could be missed, a counter that touches five places, nothing for a reviewer to enforce, and tool economy. Not one word about numbering.
That is not the skill being wrong. The session's own suite was green and its typecheck was clean. The log holds no failure, so the retro has nothing to explain. The bug only showed up in a check the session never ran.
So I ran it again and told it what finance had found, phrased as a report, no fix requested. This time it found the cause in its own code: 'I copied the order from the existing /issue handler.' It also predicted that the issue route leaks invoice numbers the same way. I checked. It does.
retro alone → "This session was smooth"
retro + failure → the cause, in its own words, plus five new candidates
Step one of the loop is therefore not 'run /retro'. It is 'run /retro and tell it what failed'.
Make it quote the moment, then drop what it cannot
Pocock's own docs say to discard any candidate you cannot trace to a specific moment of the session. The skill does not do that step for you, so I asked for the transcript moment behind each candidate.
On the second pass, four of five survived. The firm one quoted its own test: it had run accept, reject, accept, and asserted only status codes, never the number that came back. The weakest one retracted part of its claim on the spot: 'my inference, not your words.' The fifth was dropped because it described behaviour, not environment, and would have counted the first finding twice.
On the first pass it was harsher: two survivors out of five, and it corrected two claims of its own on the way. A retro that drops three of its own candidates when pushed is doing its job.
Nothing went into CLAUDE.md, and the docs say why
Both retro passes declined a steering-file line. The first said it 'would cost context on every turn and go stale'. The second said 'Keep it as a comment on nextId, not a steering-file line.'
The Claude Code memory docs back that up: Claude treats CLAUDE.md as context, not enforced configuration, and a block needs a hook. So each survivor went into the carrier that fits its kind:
- The missing guardrail became a tracked pre-commit hook that runs
pnpm typecheck && pnpm test. - The numbering slip became one vitest test in
test/numbering.test.ts: first note CN-0001, a rejected over-credit, then the next valid note must be CN-0002. - The judgement call became one line in
CODING_STANDARDS.md: take a sequence number only once every validation has passed. - The navigation gap became a doc comment on
nextId, saying it persists the counter at once.
Four files, thirty added lines, zero lines of CLAUDE.md.
The hook blocked my own commit first
The new test failed on its first run: CN-0003 where CN-0002 was due. Good. A test that passes on a tree that still holds the bug proves nothing.
Then I staged the four approved changes and committed. The hook ran the suite against the working tree, which still held the buggy handler: 1 failed, 42 passed. No commit. That is the hook doing on the first day what a CLAUDE.md line could not do on any day.
I asked the session to fix the handler, nothing else. It moved the number allocation after the validation checks, left the older invoice leak alone as told, and the commit went through.
One trap on the way: /doctor prompt-audit with no argument audited my user-level files and said nothing about the new standards file. With the file path as its argument it answered 'the file is clean', one line, nothing to change.
before: counter → validate → 201 or 409 (number spent either way)
after: validate → counter → 201 (a 409 spends nothing)
Then the re-run. A fresh session, the same request word for word. Its second command was to read CODING_STANDARDS.md, the README, the hook and the package file. Its third read the red test. It wrote the route with the number taken after validation, cited the standards line in its summary, and shipped with 37 tests passing. Finance check: 5/5. CN-0002 after a rejected request.
What this doesn't prove
One re-run is one re-run. The fresh session was handed the lesson as a red test plus a standards line, so its pass is the loop working as designed, not an independent replication.
The bug itself is not rare, but it is not guaranteed either. Two bare runs of the same task before the retro: the first came back clean, the second shipped the bug.
The hook is the only part of this that cannot be ignored. The standards line and the comment are read because the session chose to read them. On a bigger repo, I would expect that choice to go the other way more often.
The full run
The video shows the two retro outputs side by side, the trace step with each candidate's quoted moment, and the commit being refused on screen.
When a Claude Code session ships something wrong, what do you reach for first? A) a new line in CLAUDE.md B) a test C) a hook D) nothing, next session will be different
I had help editing the English of this piece. The run, the numbers and the opinions are mine.
Top comments (19)
The “session was smooth” result rings true: a session that wrote its own tests mostly checks the shape it already had in mind. What catches this kind of bug for me is a second model reviewing the diff with one narrow question: what did this change copy from nearby code, and is that order still right here? Copied ordering slips like taking the number before validating show up there far more often than in the session’s own green suite. Nice write-up, and the hook over a CLAUDE.md line is the right call.
I didn't run a second model over the diff in this test, so I can't compare the two. The closest thing was the second retro: once it knew what finance saw, it wrote 'I copied the order from the existing /issue handler' on its own and predicted that /issue leaks invoice numbers the same way, which it does. Your narrow question would have found that before the finance check did, without needing the failure first.
That second retro is the part I’d keep. Once it had a concrete symptom, it found the copied pattern and the second leak on its own. Next time it might be worth giving it the symptom you fear before anything breaks, something like asking whether an invoice number can reach a log here, and seeing if it names /issue unprompted.
I didn't try that, so I can't say what it would name. Two things from the run make me cautious: the trace step we ask for only keeps what the retro can quote from the session log, and the first retro dropped the migration check as a hypothetical for exactly that reason, so a feared symptom with no moment behind it might get traced out. The one tool that flagged /issue without being told was the prompt audit, once the standards line existed to check the code against.
That makes sense: if the trace only keeps what it can quote, a fear with no moment behind it has nothing to hold on to. The part I’m taking away is the prompt audit catching /issue once the standards line existed. Write the rule down first, and the check finally has something to compare against. Thanks for running it.
CN-0004 after a rejected request is such a good bug for this experiment, because every test can pass while finance sees the gap in the sequence immediately. Taking the counter before validation, copied from the /issue route, is exactly the kind of convention nobody writes down.
The part I keep thinking about is the retro saying the session was smooth until you told it what failed. That matches what Junfeng Zhou said in the comments: a CLAUDE.md line is advice and a test is a fact. Your four carriers (a pre-commit hook, one vitest test, a standards line and a doc comment on nextId) put each lesson where it can actually be enforced, and thirty lines with zero added to CLAUDE.md is a nice result.
The hook blocking your own commit first is the best evidence in the post. Emily Woods also raises a fair follow up: if the success response is lost and the client retries, should it get the same credit note number back?
Same answer as to Emily: I didn't test the retry case, and the loop as it stands says nothing about it since the test locks the rejected request only. The route takes a bare amount with nothing to key the request, so a retry would get a second note. It would be the next failure report to hand the retro, the way the finance report was the first.
The next case I’d add is losing the success response and retrying the same request. I’d want one credit note and the same number back, even though both requests are valid. That’s the boundary I’d carry into a coffee-ordering design exercise too like one checkout should still mean one order after a timeout.
I didn't test that case, and nothing in the run covers it: the test and the hook only lock the rejected request. The route takes a bare amount and nothing keys the request, so a retry after a lost response would mint a second note with a new number. That would be the next failure to feed the retro.
A green suite says the tests matched the implementation, not that the implementation matched the real contract. The retro should add the missing observable to the regression: a production-shaped fixture, an integration boundary, or an invariant that would have failed before the bug reached users.
That is the test the retro added: one credit note accepted, one rejected, and the next valid one must carry the next number, so the gapless sequence is the invariant. It was red on its first run and it is what the commit hook tripped on before the fix went in. The production-shaped part stayed outside: the finance check that caught the bug was a hidden script we ran by hand, and we didn't fold it into the suite in this run.
The detail that stuck with me is the handler pulling CN-0004 from the counter before validating, because it copied the order from the /issue route, and that the retro only found it once you told it what finance saw. A practical tip: add a test that asserts the returned number after a rejected request (accept, reject, accept, then check the sequence), not just status codes. That kind of invariant test catches this whole class of bug, whatever the agent copied. Did the fresh session pass 5/5 after the retro?
Yes, 5/5. The fresh session read the standards line and the red test before it wrote any code, shipped with 37 tests green, and finance got CN-0002 after the rejected request. Your tip is close to the test that went in: credit 4000 for CN-0001, reject an over-credit, credit 6000 and assert CN-0002. The session's own test had run the same sequence and checked status codes only.
Green tests say the checks passed, not that the task is finished. Count the cost of the runs that end with a bug next to the ones that pass.
I did count. The run that shipped the bug was the cheapest of the three at $0.27; the clean bare run cost $0.43 and the re-run after the retro $0.31. Both retros, both trace passes, the four applies and the fix came to about $0.45 together, so roughly one extra run bought the hook and the test.
An unwritten convention is just a bug nobody has filed yet.
The first retro went one step further: it declined to write any standards file 'until a real review miss shows one'. The line only exists because the miss happened, which is your point run forward.
A test first, since a CLAUDE.md line is advice and a test is a fact. The invariant here is a nice one: a rejected request must not change the next number.
That split held up in the run. The test failed first time, CN-0003 where CN-0002 was due, against the tree that still held the bug, and the hook refused the commit on the same failure. Nothing in a steering file can refuse a commit.