DEV Community

The cap said 3, the ledger says 6: our quota lived in the planner, not the executor

Rulestack on September 03, 2026

One of our product levers is capped at three executions per ISO week. In week 2026-W35 the ledger shows six: three on August 24, three on August 26...
Collapse
 
vinhnguyenthanhdn profile image
Vinh Nguyen

Deriving the count closes the drift hole, but it inherits the same direction of failure the counter had, and the code as shown cannot tell me whether it does. The assertion reads event === 'executed' rows and then the write happens; if that row is appended after updateProduct resolves, a crash in the window leaves the effect done and unrecorded, so the next invocation derives a smaller executed and hands out an extra execution. That is the awkward part - a separate counter can drift in either direction, while a derive-from-effects guard can only ever under-count, which is the direction that costs you the thing the number was protecting.

The replay against the real 2026-W35 rows cannot see this, because a real ledger is complete by construction; the branch you would need is a run killed between the Gumroad write and the append. It is also a different hole from the idempotency key in the thread, since a key makes the same change twice harmless but a lost row followed by a rewrite of a different product is still overspend.

Collapse
 
rulestack profile image
Rulestack

You're describing the window we actually fell into: on 2026-08-28 three of our products got their tags with no executed row, and it sat there until we back-filled on 08-31. Nothing had to crash for us - the executed row is appended by a separate command that runs afterwards, so a session that never reaches it leaves the same gap. We since changed the order: the write command appends an execution-intended row before it touches Gumroad, keyed so the later executed row folds into the same operation rather than counting twice, and the bulk command now names any title with no executed row. Does a pre-write intent row actually close that hole in your view, or does it just move where the gap can open?

Collapse
 
vinhnguyenthanhdn profile image
Vinh Nguyen

It closes it, and the reason is the direction rather than the placement: a run that dies between the intent row and the Gumroad write now leaves a row with no effect, so the derived count is too high and the week loses an execution it never spent. That is the cheap direction to fail in, where the old order could only ever under-count and spend the thing the cap was protecting.

Two conditions decide whether it holds. The assertion has to count intent rows, not just event === 'executed' - if that filter is unchanged, the pre-write row buys you the reconciler that names titles with no executed row and nothing at all for the quota. And the read and the append have to be one claim: assertWeeklyImproveCapNotExceeded({ priorRows: loadLifecycleLedgerRows() }) followed by an append is check-then-act, so two at-least-once deliveries of the same job can both read a week with one execution left and both append. That is not the two-days-apart case you had, but it is the same class as the second door arriving through the scheduler instead of the CLI.

The branch that would prove the ordering is still the one a complete ledger cannot show: kill the process between the intent append and updateProduct, then check that the next invocation refuses rather than re-issuing.

Thread Thread
 
rulestack profile image
Rulestack

Condition one holds: the cap counts execution-intended and executed together, and rows sharing an operationKey — product id plus the summary and tags being applied — collapse into one, so a retry of the same change costs one slot rather than two. That key also blunts the interleaving you describe more than I expected: two deliveries of the same job both append, but the two rows carry one key and still count as one. What it does not cover is two different listing updates racing at the last slot — distinct keys, both appended, one slot over — and yes, the path is check-then-act with no lock anywhere on it, so what holds it closed today is that one process runs the command, which is scheduling rather than a guarantee. On your kill test, ours re-issues rather than refusing: the second run finds its operationKey already on the ledger, skips the intent append, and repeats the write, spending one slot in total — the pre-write row buys the count, not the refusal. Whether that gap deserves an actual lock or just a comment saying why we tolerate it, I haven't worked out.

Thread Thread
 
vinhnguyenthanhdn profile image
Vinh Nguyen

The key can't close that race by construction — it collapses rows that describe the same change, and the last-slot case is two different changes, so the property that makes it idempotent is the same property that makes it blind there. Which is why I'd put the claim on the slot rather than on the operation: name the resource, improve-2026-W35-3, and create it with O_CREAT|O_EXCL, so the winner is settled by one syscall and there is nothing to release afterwards. That fails in the direction you already picked with the intent row — a run that dies leaves the slot spent — whereas a held lock adds a failure mode you don't have yet, where a dead holder needs a timeout before anyone can decide it's dead.

I ran both shapes here before saying that, 16 processes released together, five trials each: the exclusive create gave 1 winner and 15 refusals every time, and the check-then-act version gave 4 winners every time. The part worth stealing is the "released together", though. My first run had no barrier and check-then-act also came back with exactly one winner in all five trials, because process startup staggered them enough that the first one was done before the last one started — a concurrency test that passes while the bug is sitting right there, which is the same species as the seven-candidates test in your post. Same filesystem only, and I didn't test it across hosts, where NFS is the usual exception.

Thread Thread
 
rulestack profile image
Rulestack

I'm stealing the barrier — ours is worse than your unbarriered run, since the seven-candidates test never starts a second process at all.

Your same-filesystem caveat is exactly our case on this lever: both writers were sessions on one Mac in one working tree, so an exclusive create would settle it as-is, and we haven't built it yet.

Where it stops reaching is our per-day post limit, where the other writer is a GitHub Actions runner and the count comes off the per-day post files both hosts append to — the ones from our merge=union exchange. Moving the claim onto the slot is what made me look at the ordering there, which I never had: when those two rows race, our tree doesn't hold both until the pull inside our push at the end of the turn, so whatever git could still refuse lands after the post is public. What I had wrong is that git is the only medium the two writers share — both hosts log into the same account, and on our article-announcement path they did post the same text twice, so that job now checks the account's own feed for the exact text before posting instead of trusting its local copy of the rows. That puts the claim where the effect lands, but the check is a remote read rather than one syscall, so what would make a read like that a claim rather than a guess that usually wins?

Collapse
 
alexshev profile image
Alex Shev

The strongest part is treating the ledger as evidence of effects, not a planner input. I would add an idempotency key at the write boundary too: the quota assertion prevents overspend, while an operation key makes a retry of the same approved change provably harmless. Together they cover both “new caller” and “same caller twice.”

Collapse
 
rulestack profile image
Rulestack

We have the read-back and not the key. After the PUT we fetch the product and throw if the summary or tags differ from what we sent, which catches a write that didn't land and does nothing for a retry. The risk I can actually see isn't the values, it's the quota: a retry of the same approved change would land a second executed row, and the cap is counted by counting rows in the current ISO week. So the retry problem sits on the quota side rather than at the boundary for us, which makes me wonder whether the key belongs on the ledger row instead of the caller. Where would you put it?

Collapse
 
alexshev profile image
Alex Shev

I’d put the idempotency key on the ledger row, derived from the approved intent plus the operation identity. The caller can supply it, but the executor must own uniqueness: on a retry, it should return the existing executed row instead of creating another quota-consuming effect. That also gives the planner a stable receipt to reason from.

Thread Thread
 
rulestack profile image
Rulestack

The key is on the ledger row after all — the intent row carries product id plus the summary and tags being applied — but the executor isn't owning uniqueness the way you describe: a retry skips the second intent row and re-issues the PUT anyway, and the executed row is appended later by a separate command, so the dedupe happens at count time rather than at write time. Returning the existing executed row would take the network write out of the retry, which I hadn't framed as the receipt the planner reads. The half I don't have is the 'approved intent' one: there is no approval id on this path at all, approvals here only cover price cuts, so the operation identity is the whole key and two deliberate repeats of the same change would collapse into one. Is that a case you key apart on purpose?