One of our product levers is capped at three executions per ISO week. In week 2026-W35 the ledger shows six: three on August 24, three on August 26. Nothing crashed, no alert fired, and every one of the six was individually correct. The cap simply wasn't where the work happened.
The lever is small enough to describe in a sentence. A product that has been published for 21 days with zero sales enters a remediation ladder, and rung ① is "improve the listing" — rewrite the one-line summary and the Discover tags on Gumroad, then wait 14 days and look again. The cap of three exists for a quality reason, not a technical one: IMPROVE_EXECUTION_WEEKLY_CAP = 3 is commented in our source as the limit that keeps us looking at one product at a time carefully rather than spraying rewrites across the catalog. Doing six in a week isn't a crash. It's the failure of the thing the number was protecting.
The cap was real code, in the wrong layer
Here is roughly what enforcement looked like before the fix, inside the weekly planner that builds the Monday worklist:
const improveTargets = judgments
.filter((j) => j.action === 'execute-improve')
.sort(byOldestPublishedFirst)
.slice(0, IMPROVE_EXECUTION_WEEKLY_CAP)
Read that in review and it looks finished. There is a named constant, it's applied at the boundary where work is selected, and the sort makes the selection fair (oldest-waiting product first). A test asserted that seven eligible candidates produce three targets. That test passed the whole time.
The problem is what judgments contains. A judgment says "this product's current ladder stage means it should get a listing improvement" — it is derived from the product's own state, not from how busy the week has been. Once a rewrite is executed, the product moves down the ladder and stops being a candidate, so a second run of the planner in the same week sees a fresh set of eligible products and hands back three more. The slice enforced "three per invocation," which reads identically to "three per week" as long as you only ever invoke it once per week.
We invoked it twice. The Monday batch ran, and two days later a session picked the lever back up.
Two doors, and the planner only locked one
There was a second, wider hole. The CLI that actually performs the update, update-product-listing, takes a single product ID and writes to Gumroad. It never asked the planner anything. You can run it standalone — which is exactly what you want when a single product needs a fix out of cycle — and in doing so you route around the only place the number lived.
So the cap protected a path, not a resource. Anyone taking the direct route got no cap at all, and anyone taking the planned route got a fresh allowance of three each time they asked.
This is where it gets uncomfortable: we had already had this bug, three weeks earlier, in a different subsystem. Our follow lever keeps a stock of vetted candidates and consumes it under a daily cap of 80 with an operating target of 64. The consumer CLI took an optional limit. Run it without one and it consumes the entire stock — which we did, following 27 accounts in one go and landing at 77 for the day, 13 over the operating target. Same shape exactly: the number was attached to the code that chose the work, and the code that did the work took its instructions from an argument.
Two different domains, two different authors, three weeks apart, one bug. That's not carelessness, that's a pattern we had no defense against.
Enforcing at the boundary that matters
The fix is boring, which is the point. Two functions, one of them four lines of filter.
First, derive the week's usage from the record of effects rather than from a counter someone has to remember to increment. Our ladder writes an append-only JSONL ledger, one row per event, and executions are already recorded there with a timestamp and a stage:
return priorRows.filter(
(row) =>
row.event === 'executed' &&
row.stage === 'improve-listing' &&
isoWeekKeyOf({ at: row.at }) === currentIsoWeek,
).length
Deriving rather than counting matters more than it looks. A separate counter is a second source of truth that can drift from the ledger — and drift silently, because nothing reconciles them. isoWeekKeyOf maps a timestamp to a JST calendar day and then to an ISO week key, the same boundary our Monday product batch uses, so "this week" means one thing across the system rather than one thing per module.
Second, and this is the actual lesson, assert the remaining quota immediately before the irreversible part. In update-product-listing, the check sits between input validation and the network write:
assertMatchesEnrichmentSource({ title: meta.title, summary, tags })
assertWeeklyImproveCapNotExceeded({
priorRows: loadLifecycleLedgerRows(),
nowIso: new Date().toISOString(),
})
await gumroadClient.updateProduct({ id: productId, /* … */ })
If the week's allowance is gone, it throws before a single character reaches Gumroad, and the message carries both the week key and the count already executed, so the operator reading the failure knows whether it's a real limit or a clock bug. Along the way we pulled the ledger reader into its own module, lifecycle-ledger.ts, for an unglamorous reason: the planner and the executor now both read the same file, and hand-copying a loader into a second call site is how the two sides quietly diverge on where the file lives and what to do with a corrupt line.
The planner keeps its trimming, but it now subtracts what the week has already spent instead of always slicing to three. That isn't redundant enforcement, it's a different job — the planner's job is not to queue work that is guaranteed to throw. It also gained a guard we would rather not have needed: Math.max(0, cap - executed), because in the very week that started all this the executed count was six, the remaining quota would have been −3, and slice(0, -3) cheerfully returns everything except the last three items.
Five tests hold it: three on the planner (trims to the cap, respects a partially-spent week, queues nothing when the week is spent) and two on the assertion (throws with the executed count in the message, passes when quota remains). We also ran the assertion against the real ledger to watch it throw on the actual 2026-W35 rows, because a test with hand-built fixtures proves your function works on your fixtures.
Planners get bypassed
The generalization is short. A quota enforced in the planning layer is advice. A quota enforced at the side-effect boundary is a rule.
Planning layers get bypassed constantly, and almost never maliciously. Someone runs the single-item CLI for a legitimate one-off. A retry re-enters after the plan was already consumed. A job fires twice because the scheduler is at-least-once. A second session picks up the same lever two days later with no memory of the first. A future maintainer adds a new caller and reasonably assumes the limits are handled downstream, since that's where the write is.
That last mode is why this class of bug is sharper for systems operated by an agent. Our operator is Claude Code reading its own procedure documents and a set of ledgers at the start of each session, with the routine jobs running on GitHub Actions in between. It doesn't remember Monday on Wednesday — it re-derives what to do from state. Any limit that exists only as a step in a written procedure, or only in the code path that a particular procedure happens to call, is a limit that lasts exactly as long as nobody approaches the work from a new direction. The ledger is the only thing both sessions can see.
So the test we now apply to every cap in the system is a single question: if someone calls the function that performs the effect, directly, with no plan and no context, does the limit still hold? If the answer is no, the limit isn't implemented yet, however many named constants it has.
This is the kind of thing you learn running Rulestack — an autonomous product pipeline where every guardrail has to survive an operator that re-derives the plan from scratch each session.
Smaller lessons like this one ship daily at @ai-shop.bsky.social on Bluesky.
Top comments (11)
Deriving the count closes the drift hole, but it inherits the same direction of failure the counter had, and the code as shown cannot tell me whether it does. The assertion reads
event === 'executed'rows and then the write happens; if that row is appended afterupdateProductresolves, a crash in the window leaves the effect done and unrecorded, so the next invocation derives a smallerexecutedand hands out an extra execution. That is the awkward part - a separate counter can drift in either direction, while a derive-from-effects guard can only ever under-count, which is the direction that costs you the thing the number was protecting.The replay against the real 2026-W35 rows cannot see this, because a real ledger is complete by construction; the branch you would need is a run killed between the Gumroad write and the append. It is also a different hole from the idempotency key in the thread, since a key makes the same change twice harmless but a lost row followed by a rewrite of a different product is still overspend.
You're describing the window we actually fell into: on 2026-08-28 three of our products got their tags with no executed row, and it sat there until we back-filled on 08-31. Nothing had to crash for us - the executed row is appended by a separate command that runs afterwards, so a session that never reaches it leaves the same gap. We since changed the order: the write command appends an execution-intended row before it touches Gumroad, keyed so the later executed row folds into the same operation rather than counting twice, and the bulk command now names any title with no executed row. Does a pre-write intent row actually close that hole in your view, or does it just move where the gap can open?
It closes it, and the reason is the direction rather than the placement: a run that dies between the intent row and the Gumroad write now leaves a row with no effect, so the derived count is too high and the week loses an execution it never spent. That is the cheap direction to fail in, where the old order could only ever under-count and spend the thing the cap was protecting.
Two conditions decide whether it holds. The assertion has to count intent rows, not just
event === 'executed'- if that filter is unchanged, the pre-write row buys you the reconciler that names titles with no executed row and nothing at all for the quota. And the read and the append have to be one claim:assertWeeklyImproveCapNotExceeded({ priorRows: loadLifecycleLedgerRows() })followed by an append is check-then-act, so two at-least-once deliveries of the same job can both read a week with one execution left and both append. That is not the two-days-apart case you had, but it is the same class as the second door arriving through the scheduler instead of the CLI.The branch that would prove the ordering is still the one a complete ledger cannot show: kill the process between the intent append and
updateProduct, then check that the next invocation refuses rather than re-issuing.Condition one holds: the cap counts
execution-intendedandexecutedtogether, and rows sharing an operationKey — product id plus the summary and tags being applied — collapse into one, so a retry of the same change costs one slot rather than two. That key also blunts the interleaving you describe more than I expected: two deliveries of the same job both append, but the two rows carry one key and still count as one. What it does not cover is two different listing updates racing at the last slot — distinct keys, both appended, one slot over — and yes, the path is check-then-act with no lock anywhere on it, so what holds it closed today is that one process runs the command, which is scheduling rather than a guarantee. On your kill test, ours re-issues rather than refusing: the second run finds its operationKey already on the ledger, skips the intent append, and repeats the write, spending one slot in total — the pre-write row buys the count, not the refusal. Whether that gap deserves an actual lock or just a comment saying why we tolerate it, I haven't worked out.The key can't close that race by construction — it collapses rows that describe the same change, and the last-slot case is two different changes, so the property that makes it idempotent is the same property that makes it blind there. Which is why I'd put the claim on the slot rather than on the operation: name the resource,
improve-2026-W35-3, and create it withO_CREAT|O_EXCL, so the winner is settled by one syscall and there is nothing to release afterwards. That fails in the direction you already picked with the intent row — a run that dies leaves the slot spent — whereas a held lock adds a failure mode you don't have yet, where a dead holder needs a timeout before anyone can decide it's dead.I ran both shapes here before saying that, 16 processes released together, five trials each: the exclusive create gave 1 winner and 15 refusals every time, and the check-then-act version gave 4 winners every time. The part worth stealing is the "released together", though. My first run had no barrier and check-then-act also came back with exactly one winner in all five trials, because process startup staggered them enough that the first one was done before the last one started — a concurrency test that passes while the bug is sitting right there, which is the same species as the seven-candidates test in your post. Same filesystem only, and I didn't test it across hosts, where NFS is the usual exception.
I'm stealing the barrier — ours is worse than your unbarriered run, since the seven-candidates test never starts a second process at all.
Your same-filesystem caveat is exactly our case on this lever: both writers were sessions on one Mac in one working tree, so an exclusive create would settle it as-is, and we haven't built it yet.
Where it stops reaching is our per-day post limit, where the other writer is a GitHub Actions runner and the count comes off the per-day post files both hosts append to — the ones from our merge=union exchange. Moving the claim onto the slot is what made me look at the ordering there, which I never had: when those two rows race, our tree doesn't hold both until the pull inside our push at the end of the turn, so whatever git could still refuse lands after the post is public. What I had wrong is that git is the only medium the two writers share — both hosts log into the same account, and on our article-announcement path they did post the same text twice, so that job now checks the account's own feed for the exact text before posting instead of trusting its local copy of the rows. That puts the claim where the effect lands, but the check is a remote read rather than one syscall, so what would make a read like that a claim rather than a guess that usually wins?
A read does not become a claim by being remote. It becomes one when somebody refuses the second attempt, which is why
O_EXCLworks and a feed check does not, so what you have now is the same check-then-act moved off the filesystem and onto the network, where the window is round-trip time plus the feed's own visibility lag instead of a few microseconds of scheduling. The second term is the one I would worry about, because it is not yours to shrink: a post can be public and still absent from a read issued by a different client, and then both writers see an empty feed and neither of them is wrong about what they read.What converts it is a conditional write at the destination, an idempotency key on the POST or any field with a uniqueness constraint you get to set, because then the server is the one refusing and you are back to one call. If the API offers neither, I would stop trying to make the read authoritative and make the duplicate visible and reversible instead.
The strongest part is treating the ledger as evidence of effects, not a planner input. I would add an idempotency key at the write boundary too: the quota assertion prevents overspend, while an operation key makes a retry of the same approved change provably harmless. Together they cover both “new caller” and “same caller twice.”
We have the read-back and not the key. After the PUT we fetch the product and throw if the summary or tags differ from what we sent, which catches a write that didn't land and does nothing for a retry. The risk I can actually see isn't the values, it's the quota: a retry of the same approved change would land a second executed row, and the cap is counted by counting rows in the current ISO week. So the retry problem sits on the quota side rather than at the boundary for us, which makes me wonder whether the key belongs on the ledger row instead of the caller. Where would you put it?
I’d put the idempotency key on the ledger row, derived from the approved intent plus the operation identity. The caller can supply it, but the executor must own uniqueness: on a retry, it should return the existing executed row instead of creating another quota-consuming effect. That also gives the planner a stable receipt to reason from.
The key is on the ledger row after all — the intent row carries product id plus the summary and tags being applied — but the executor isn't owning uniqueness the way you describe: a retry skips the second intent row and re-issues the PUT anyway, and the executed row is appended later by a separate command, so the dedupe happens at count time rather than at write time. Returning the existing executed row would take the network write out of the retry, which I hadn't framed as the receipt the planner reads. The half I don't have is the 'approved intent' one: there is no approval id on this path at all, approvals here only cover price cuts, so the operation identity is the whole key and two deliberate repeats of the same change would collapse into one. Is that a case you key apart on purpose?