I gave a plain Claude Code session a task on a small invoicing API: add credit notes, numbered CN-0001, CN-0002, in sequence. It came back with 42 ...
For further actions, you may consider blocking this person and/or reporting abuse
The “session was smooth” result rings true: a session that wrote its own tests mostly checks the shape it already had in mind. What catches this kind of bug for me is a second model reviewing the diff with one narrow question: what did this change copy from nearby code, and is that order still right here? Copied ordering slips like taking the number before validating show up there far more often than in the session’s own green suite. Nice write-up, and the hook over a CLAUDE.md line is the right call.
I didn't run a second model over the diff in this test, so I can't compare the two. The closest thing was the second retro: once it knew what finance saw, it wrote 'I copied the order from the existing /issue handler' on its own and predicted that /issue leaks invoice numbers the same way, which it does. Your narrow question would have found that before the finance check did, without needing the failure first.
That second retro is the part I’d keep. Once it had a concrete symptom, it found the copied pattern and the second leak on its own. Next time it might be worth giving it the symptom you fear before anything breaks, something like asking whether an invoice number can reach a log here, and seeing if it names /issue unprompted.
I didn't try that, so I can't say what it would name. Two things from the run make me cautious: the trace step we ask for only keeps what the retro can quote from the session log, and the first retro dropped the migration check as a hypothetical for exactly that reason, so a feared symptom with no moment behind it might get traced out. The one tool that flagged /issue without being told was the prompt audit, once the standards line existed to check the code against.
That makes sense: if the trace only keeps what it can quote, a fear with no moment behind it has nothing to hold on to. The part I’m taking away is the prompt audit catching /issue once the standards line existed. Write the rule down first, and the check finally has something to compare against. Thanks for running it.
CN-0004 after a rejected request is such a good bug for this experiment, because every test can pass while finance sees the gap in the sequence immediately. Taking the counter before validation, copied from the /issue route, is exactly the kind of convention nobody writes down.
The part I keep thinking about is the retro saying the session was smooth until you told it what failed. That matches what Junfeng Zhou said in the comments: a CLAUDE.md line is advice and a test is a fact. Your four carriers (a pre-commit hook, one vitest test, a standards line and a doc comment on nextId) put each lesson where it can actually be enforced, and thirty lines with zero added to CLAUDE.md is a nice result.
The hook blocking your own commit first is the best evidence in the post. Emily Woods also raises a fair follow up: if the success response is lost and the client retries, should it get the same credit note number back?
Same answer as to Emily: I didn't test the retry case, and the loop as it stands says nothing about it since the test locks the rejected request only. The route takes a bare amount with nothing to key the request, so a retry would get a second note. It would be the next failure report to hand the retro, the way the finance report was the first.
The next case I’d add is losing the success response and retrying the same request. I’d want one credit note and the same number back, even though both requests are valid. That’s the boundary I’d carry into a coffee-ordering design exercise too like one checkout should still mean one order after a timeout.
I didn't test that case, and nothing in the run covers it: the test and the hook only lock the rejected request. The route takes a bare amount and nothing keys the request, so a retry after a lost response would mint a second note with a new number. That would be the next failure to feed the retro.
A green suite says the tests matched the implementation, not that the implementation matched the real contract. The retro should add the missing observable to the regression: a production-shaped fixture, an integration boundary, or an invariant that would have failed before the bug reached users.
That is the test the retro added: one credit note accepted, one rejected, and the next valid one must carry the next number, so the gapless sequence is the invariant. It was red on its first run and it is what the commit hook tripped on before the fix went in. The production-shaped part stayed outside: the finance check that caught the bug was a hidden script we ran by hand, and we didn't fold it into the suite in this run.
The detail that stuck with me is the handler pulling CN-0004 from the counter before validating, because it copied the order from the /issue route, and that the retro only found it once you told it what finance saw. A practical tip: add a test that asserts the returned number after a rejected request (accept, reject, accept, then check the sequence), not just status codes. That kind of invariant test catches this whole class of bug, whatever the agent copied. Did the fresh session pass 5/5 after the retro?
Yes, 5/5. The fresh session read the standards line and the red test before it wrote any code, shipped with 37 tests green, and finance got CN-0002 after the rejected request. Your tip is close to the test that went in: credit 4000 for CN-0001, reject an over-credit, credit 6000 and assert CN-0002. The session's own test had run the same sequence and checked status codes only.
Green tests say the checks passed, not that the task is finished. Count the cost of the runs that end with a bug next to the ones that pass.
I did count. The run that shipped the bug was the cheapest of the three at $0.27; the clean bare run cost $0.43 and the re-run after the retro $0.31. Both retros, both trace passes, the four applies and the fix came to about $0.45 together, so roughly one extra run bought the hook and the test.
An unwritten convention is just a bug nobody has filed yet.
The first retro went one step further: it declined to write any standards file 'until a real review miss shows one'. The line only exists because the miss happened, which is your point run forward.
A test first, since a CLAUDE.md line is advice and a test is a fact. The invariant here is a nice one: a rejected request must not change the next number.
That split held up in the run. The test failed first time, CN-0003 where CN-0002 was due, against the tree that still held the bug, and the hook refused the commit on the same failure. Nothing in a steering file can refuse a commit.