Every network call can fail in three ways: it fails, it succeeds, or it succeeds and you never find out. The third case is where most production incidents live.
Junior instinct: "Request timed out, retry it."
Senior question: "What if the first one actually went through?"
The core problem
Client Server DB
| --- POST /charge --> | |
| | --- INSERT charge -> |
| | <------- OK -------- |
| X (response lost) | |
| --- POST /charge --> | (retry) |
| | --- INSERT charge -> | <- duplicate
The client can't tell "never arrived" from "arrived, reply lost." Retrying blindly means double charges, duplicate orders, double emails.
Fix 1: Idempotency keys
The client generates a unique key per logical operation and sends it on every attempt. The server stores the result and replays it.
Client Server
| POST /charge |
| Idempotency-Key: abc-123 -> | not seen: process, store result
| X (response lost) |
| POST /charge |
| Idempotency-Key: abc-123 -> | seen: return stored result, no new charge
app.post("/charge", async (req, res) => {
const key = req.get("Idempotency-Key");
if (!key) return res.status(400).json({ error: "Idempotency-Key required" });
// Atomic claim: unique constraint on key
const claimed = await db.idempotency.insertIfAbsent({
key, status: "in_progress", requestHash: hash(req.body),
});
if (!claimed) {
const existing = await db.idempotency.get(key);
if (existing.requestHash !== hash(req.body))
return res.status(422).json({ error: "Key reused with different payload" });
if (existing.status === "in_progress")
return res.status(409).json({ error: "Request in progress" });
return res.status(existing.code).json(existing.response); // replay
}
const result = await chargeCard(req.body);
await db.idempotency.complete(key, 200, result);
res.json(result);
});
Details that matter:
- Make the claim atomic (unique constraint, not check-then-insert).
- Hash the payload. Same key with a different body is a client bug, so reject it.
- Handle the in-progress state so concurrent retries don't race.
- Set a TTL (24-72h is common).
Fix 2: Retry correctly
Retries without backoff turn a blip into an outage (the retry storm).
Attempt: 1 2 3 4 5
Wait: 0 -> 1s -> 2s ------> 4s -----------> 8s (exponential)
+-jitter: random within each window
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function retry(fn, { retries = 5, base = 200, cap = 10_000 } = {}) {
for (let i = 0; ; i++) {
try { return await fn(); }
catch (err) {
if (i >= retries || !isRetryable(err)) throw err;
const backoff = Math.min(cap, base * 2 ** i);
await sleep(Math.random() * backoff); // full jitter
}
}
}
- Jitter is not optional. Without it, every client retries in lockstep.
- Only retry transient errors (timeouts, 429, 503), never 400/401/404.
- Respect
Retry-After. Put a retry budget at the service level.
Fix 3: The dual-write problem
await db.save(order); // succeeds
await queue.publish(event); // crashes here -> DB and queue disagree
You can't make two systems commit atomically. Use the transactional outbox:
+---------- one DB transaction ----------+
| INSERT order INSERT outbox_event |
+-----------------------------------------+
|
relay polls outbox -> publishes to queue -> marks sent
This gives at-least-once delivery, so consumers must be idempotent too (dedupe by event ID).
What I'd standardize on
| Layer | Rule |
|---|---|
| Write endpoints | Require Idempotency-Key
|
| Clients | Exponential backoff + full jitter + cap |
| Messaging | Outbox + idempotent consumers |
| Semantics | Assume at-least-once, design for it |
Takeaway
"Exactly-once" is mostly a myth. What you actually build is at-least-once delivery plus idempotent processing, which behaves like exactly-once from the outside.
Design every write as if it will be sent twice, because eventually it will be.
How does your team handle idempotency? Keys, natural dedupe, or hope? Let me know in the comments. 👇
Top comments (3)
The claim step and the finish step are separated by
chargeCard, so a process that dies in between leaves the row atin_progressfor good. Every retry then gets the 409 until the TTL expires, and the customer may or may not have been charged. Two things close that: pass the same key to the card provider so a second attempt is deduplicated there, and let a retry take over a claim older than the longest the charge can run.On the outbox, the dedupe table's TTL has to outlast the longest redelivery delay. A consumer that was down for a weekend and sees an event again after a 72h key window has expired will process it twice.
Does your claim row carry a timestamp for takeover, or does it only ever move to done?
The in-progress state is the part most idempotency writeups skip, and it matters. One gap in the handler as written: if the process dies after the claim but before complete(), the key stays in_progress and every retry gets 409 until the TTL. A lease timestamp on the claim, so a stale in-progress row can be taken over after a bounded time, closes it. Also worth deciding whether a failed charge stores its error for replay or releases the key, since retrying a deterministic failure with the same key should not behave differently from a transient one.
The third failure mode — succeeds, and you never find out — is the right place to start, because it reframes idempotency from a nice-to-have into the only honest response to an unknowable outcome. One detail worth adding to the key pattern: storing the key is only half of it; the stored response has to be returned on the retry, so the client gets the original result instead of either a duplicate or a bare 'already processed' error it cannot reconcile. Key expiry matters for the same reason — too short and a late retry duplicates, too long and you are storing responses forever.