In the counting-houses of old Venice, a merchant's quiet terror was not the thief in the night. It was the honest scribe who wrote the same payment...
For further actions, you may consider blocking this person and/or reporting abuse
The lease is the right fix, and it opens the one race nobody draws in the diagram: the original handler isn't always dead when the lease expires — sometimes it's just slow. The retry takes over, both run, and now the double charge is back wearing a lease's clothes. The guard that closes it is a fencing token: every claim increments a counter on the record, and the "done" write is conditional on the token still being yours. The slow owner's completion write then fails the condition, and its side effect has to check the token before it commits too. Lease decides who may work; fencing decides whose work counts.
Fair catch... and honestly a hole in my own article. I wrote that a retry finding in_progress “may safely re-attempt”, and a slow owner that isn’t dead proves that isn’t always safe. Your fencing token closes it: every claim bumps a version on the record, and the “done” write carries a condition on that version, so the slow handler’s completion fails. The side effect has to check it too. Where the payment provider can’t see my token, the idempotency key I pass downstream is the last fence. “Lease decides who may work, fencing decides whose work counts” is the best summary of this I’ve read. I’ll fix that line in the article and credit you in the follow-up 🙂
Appreciate the fair catch on your side too — "may safely re-attempt" was always load-bearing. One concrete detail worth adding: the version bump has to be a single conditional write that both takes the lease and returns the new token atomically, otherwise two retries can each believe they own the newest token. I once shipped exactly that bug by reading the version and writing it in two steps. Should the fence live in DynamoDB or travel with the payment provider?
Thanks for that detail... a read-then-write is the classic way to get this wrong, and your two-retries scenario is exactly how it breaks. In DynamoDB the claim has to be one
UpdateItem, with the condition checked and the version bumped in the same call:The new version comes back from that same call, so there is no gap for a second retry to slip into. To your question, I'd say both, because they guard different things. DynamoDB is the fence for my own ledger: it decides who may work and whose "done" write counts. But a slow owner that has lost its lease can still call the payment API, and my table can't stop that. So the idempotency key sent to the provider, the same one on every attempt, is the only fence on the charge itself. A token travelling to the provider would be ideal, but most payment APIs I know of only accept an idempotency key, not a fencing token, so that is the one I'd rely on 🙂
That single-UpdateItem claim is exactly right — the version bump inside the same conditional write closes the gap I was worried about in the two-call version. And 'both' is the cleaner mental model: the DynamoDB condition fences who may do the work, the provider idempotency key fences the charge itself, and neither covers the other's failure. The case I'd still test: lease expiry mid-flight — a slow first attempt whose lease lapses while it's still charging, then a retry that legitimately claims the lease. Does your version bump catch that, or does the attempt need a heartbeat?
Great test case... and no, the version bump doesn’t stop the charge itself. It only makes the slow attempt’s “done” write fail afterwards. While that first attempt is still charging, the retry can legitimately claim the lease, and both reach the provider. What saves you there is the same idempotency key on every attempt: the provider sees the second request as a repeat, so you get one charge, not two. The version bump then makes sure only the new owner can mark it done.
A heartbeat helps, but only partly. Renew the lease with a conditional write on the version, and if the renewal fails, abort before the charge. But checking the token right before the side effect is still check-then-act, so the provider key stays the real guard.
On Lambda there’s an easier lever too. The platform kills the invocation at its timeout, so if the lease is longer than the function timeout plus a margin, a “slow but alive” owner can’t outlive its lease. What can still slip through is a request already in flight at the provider when the invocation dies, and the idempotency key covers that. I haven’t load-tested this exact scenario, so treat it as the design I’d choose, not a measured result 🙂
The 'at-least-once twice over' framing is the part worth keeping — either delivery guarantee alone would be manageable, but stacked together they guarantee the phantom charge eventually, with clean logs the whole time. Recording the work as in progress before the side effect, and passing the same key downstream to whoever actually takes the money, is the right pair of moves, because a dedupe check that stops at your own handler still leaves the payment call itself unprotected. The remaining gap is the in-progress record that never reaches done: that entry needs an expiry and an investigation path, or the cure becomes a permanently blocked legitimate charge.
Thank you... and you’ve put your finger on the gap that bites hardest in production. An “in progress” record that never reaches “done” turns a safety net into a permanent block. The fix I’d reach for is a lease: the record carries an expiry, a retry after it expires is allowed to take over, and the same key goes downstream so that retry can’t double-charge. On top of that, an alarm on entries older than the lease gives you the investigation path. Might be worth a follow-up post 🙂