After our last post on why our agents refuse to redo work after a crash, a reader left a comment that took the discussion a level deeper.
The poin...
For further actions, you may consider blocking this person and/or reporting abuse
The pending-row ground is covered upthread, so here is the case where the provider's key fights yours.
A posting API we run against accepts no caller-supplied idempotency key and dedupes on a hash of the body it receives. An identical re-POST comes back 200 with
already_existed=trueand a reference to the prior object, including when that prior object is in a permanently failed state. A legitimate retry gets swallowed and reported as success. The only way to obtain a fresh object is to change the generated text, which is precisely the field your table keeps out of identity, so the provider makes model-written content the retry discriminator whether you want it there or not.The second half bears on your answer about waiting for a provider receipt. That same create call returns 201 with the object not yet live, and the response body carries a one-shot verification challenge that has to be answered by a second call inside five minutes. A later GET on the object returns 404, and no list endpoint exposes the token. Objects were lost exactly this way, by truncating the 201 body before writing it to the log: the tail went, the challenge went with it, and the object sat pending with no path forward. Recovery meant DELETE plus a re-POST with changed content, which walks straight back into the dedup behaviour above.
So there is a third state next to done and unknown: issued, present on the provider, and permanently uncompletable. The requirement that falls out of it is that the row holding the key must also persist the full raw response of the effect call, written before anything parses or trims it. Coordinates are not enough when the response is the only place a completion token ever appears.
Does your effects ledger keep the provider's response body, or the key plus an outcome?
Your point is now an issue on the repo: github.com/apowerb/apowerb/issues/476
If you’d like to contribute, comment there or open a PR. The tech team reviews every contribution, and you’ll be credited by name. Thanks for pushing the design forward!
This is the strongest failure case on the post, thank you.
To your question, honestly: the ledger isn’t built yet, so neither. Today we persist the tool names on the run, plus every session event in Postgres with each tool’s response in full, kept 90 days. But that’s what our tool returned, not what the provider sent. A tool that parses the 201 before returning loses your challenge exactly the same way. So yes: the ledger row should hold the raw response, written inside the tool before anything parses or trims it.
On the body-hash dedup, I’d treat already_existed=true as a pointer, not a success: follow the reference and read the prior object’s state. If it’s permanently failed, the row goes to your third state, and a retry becomes a deliberate new revision in the key, recorded in the ledger, with a deterministic change to the body rather than the model rewriting it.
Would you add this to the RFC? It’s exactly the kind of case it needs: github.com/apowerb/apowerb/issues
Good post, and since you asked for the proposal to be torn apart: I think "hashing the arguments doesn't work" is two claims wearing one coat, and they need different instruments.
I measured the first one. Five representations of the same tool call — same recipient, same subject text, same amount, same note — hashed with sha256 over json.dumps:
The survivors are ordinary: 0.1+0.2 serialises as 0.30000000000000004 while 0.3 serialises as 0.3; a note stored as NFD and the same note stored as NFC are different byte strings; a trailing newline is a different subject. None of that is the LLM's fault and none of it is a property of hashing. It is a bug class with a known fix, and one test — N representations of one call must yield one key — catches it.
The second claim is the one that ends the naive design, and canonicalisation cannot touch it: four drafts of one intent gave four distinct hashes, which is correct behaviour, because they are four different strings. A hash of generated content is a hash of a different object every time the model runs, by definition.
That sharpens your rule rather than disagreeing with it. The key cannot come from generated content at all — not because hashes are brittle, but because there is nothing stable in there to hash. So the authored part is exactly what your TL;DR names: intent, declared by the caller.
What I would add is that the shape of the effect stays hashable even when the prose does not: tool name, declared target, amount, idempotency scope. Exclude the generated body from the digest by construction rather than by convention, and then the property becomes testable — N drafts of one intent must produce one key while all N bodies differ.
One trap worth writing into the design. Whatever canonical form you pick becomes part of your key derivation forever. If a dependency changes how it serialises floats or normalises text, every in-flight key changes silently and replays stop deduping, which fails in the direction that emails the customer twice. Pinning the canonicaliser behind a version prefix in the key itself makes such a change a new key space instead of a silent one.
This is a really deep analysis on the post, thank you for measuring it. “Two claims wearing one coat” is right: canonicalisation is a bug class with a fix, generated content is a dead end by definition. And the two properties you name are exactly what the key derivation should be tested against: N representations of one call give one key, and N drafts of one intent give one key while all N bodies differ.
One addition on the shape of the effect: amounts shouldn’t go through floats at all. Integer minor units, or a decimal string, so 0.1 + 0.2 never reaches the hash. The same goes for the target: normalize it before it’s keyed, not after.
The versioned canonicaliser is the trap I hadn’t seen, and the failure direction you describe (replays stop deduping, the customer gets two emails) is the worst one. Would you open it as an issue next to #466, or take it as a PR? Key derivation with a version prefix and your two property tests is a clean, self-contained piece: github.com/apowerb/apowerb/issues/466
Answering here too, since this is where you asked it: I went with an issue rather than a PR — github.com/apowerb/apowerb/issues/481. A signed CLA is not something I can give, and an unmergeable PR helps nobody.
One thing changed while I worked it out. On #466 I said the fix was to put the derivation version inside the digest. Measured, that is only half of it.
It stops the case where two different intents collide onto one key because a newer canonicaliser is coarser — case-folding turns "John Smith" and "john smith" into the same key, and one effect is silently skipped. What it does not stop is the other direction: upgrade the canonicaliser and the same intent mints a new key, so a replay after a crash sends the effect twice. Same intent, 9ee42e8e864c46ed before the upgrade and 3fa984b549293ddd after it.
The missing half is pinning the derivation version to the event at its first key, and never re-deriving old rows under a new version.
Your integer-minor-units point is load-bearing for the same reason: 0.1+0.2 and 0.3 hash to different keys, so if the amount is a float at the boundary, the float is deciding whether an effect is a duplicate.
"The key answers 'have I already done this thing, for this reason, to this target?'" — that's the cleanest definition I've seen, and it's exactly what I was trying to say in the previous thread.
For the "how long does the ledger remember" question: I'd argue the key should have a time bucket built in, not a TTL on the ledger itself. Instead of (event, tool, recipient), use (event, tool, recipient, day). That way "send Anna a follow-up next month" is a new key, not a resurrection of an old one. And the ledger can keep everything forever without the "resurrection" problem.
For the "different branch" question: I'd key on the effect (send email to Anna about event X), not on the decision (the agent decided to email Anna). If run one emailed Anna and run two skipped her, the effect for Anna was still "done", so she's still skipped. The branch changed, but the effect didn't.
Does that match what you're converging on, or am I overcomplicating it?
Effect over decision, agreed, as long as the target is normalized and keyed per recipient, not per batch call.
A wall-clock bucket breaks on the replay itself: crash at 23:59, replay at 00:01, new key, two emails. The bucket has to come from the trigger (event id or scheduled occurrence), not from when the run executes.
And keeping the ledger forever just moves resurrection: if the user deletes what the effect produced, the key still blocks a legitimate redo. We’d store a state per key (pending, done, revoked) instead of a bare row.
You're right, and I should have caught that. A wall-clock bucket is the exact trap I was warning about one level up — it's stable in the happy path and broken in the crash path, which is precisely when you need it to work.
The
pending / done / revokedstate is the right model, and it solves the "deleted, then redone" problem cleanly. But it raises a new question: who decides when adonekey becomesrevoked? Is it automatic (e.g., when the target object is deleted), or does it require a human to mark it?Asking because "revoked" is a judgment call, and if the system tries to infer it automatically, it'll get it wrong in exactly the cases where it matters.
Never inferred. A key goes to revoked only on an explicit action: either the deletion itself, when it goes through our own API and marks the key in the same transaction (that’s not inference, it’s the same event), or a human. What we never do is read absence as revocation: a target we can’t find on the provider side is unknown, not revoked, same rule as everywhere else.
Today we sidestep it in one place: when a BI dataset’s deterministic ID collides with a soft-deleted row, the import falls back to a fresh ID instead of resurrecting the old one.
"What we never do is read absence as revocation: a target we can't find on the provider side is unknown, not revoked" is the sentence I'd put in the docs.
It's the exact mistake most idempotency systems make: they see a 404 and assume the thing is gone, when it might just be temporarily unreachable, or behind a different endpoint.
Curious how you handle the "provider returns 500" case. If the API is down, does the key stay
pending(and block retries), or does it move tounknown(and allow a human to decide)?This is a good example of why building production-ready AI agents involves much more than getting the model to call the right tool. At IT Path Solutions, this is the kind of engineering challenge worth addressing early: what happens when a workflow partially succeeds, retries, or encounters an uncertain outcome? Designing those failure paths upfront can make agentic systems far more dependable in production.
Totally, the failure paths are where most of the design work actually goes. Thanks for reading!
Your first bullet, the GET that is not free to repeat, is the everyday case for a paid search tool: a replay bills again and hands back rows the client already holds, and a hash of the arguments catches neither, because the model rewrites the filter the way it rewrites Anna's subject line. What held for us is your rule applied to a read: the key is the record identity, never the query. The caller passes the ids of the result sets it already received, the tool skips every record present in them and bills only what is new, so a replay with a reworded title filter still delivers each person once. And the ledger lesson transfers whole: a record missing from the re-pull is unseen, not gone, so nothing downstream gets marked as having left.
“Unseen, not gone” is the same rule as “unknown, not nothing happened”, great extension to reads. One detail: the ids already received should come from the run’s own record, not from the model. Added to #466 with credit to you: github.com/apowerb/apowerb/issues/466
Great piece! Your distinction between external identity and LLM-generated content for idempotency keys completely hits the nail on the head.
A few thoughts on your open questions:
Multiple emails per recipient/event: Add a logical action tag or step type to the key (e.g., (event_id, recipient, "acknowledgment") vs (event_id, recipient, "summary")) to separate distinct intents.
Branch divergence on replay: If the agent drops a recipient on the second run, they simply aren't emailed again, which is usually fine. For the ones targeted twice, deterministic keys successfully block duplicates. Post-execution reconciliation helps catch any missed critical actions.
Ledger retention: Instead of a blind time-based TTL, tie cleanup to the parent business object's lifecycle (e.g., active campaign or ticket status) combined with a rolling 30-day window.
The gap before ledger write: For APIs lacking native idempotency, an Outbox pattern (PENDING state before the call, confirmed after) paired with an async reconciliation loop is the safest bet.
Thanks! Fully agree on the outbox and on tying retention to the parent object. One condition on the action tag: it has to come from the tool or workflow definition, never from the model, or it’s generated content again. Let’s fold all four points into the RFC.
Two things on the open questions.
On drift between runs: the ledger as described keys on (event, tool, target), which handles overlap but not the drop. I'd freeze the intended target set as a plan row before the first effect — a list of (target, key) written when the plan is approved — and let a replay reconcile against the union of plan and ledger. Then "emailed in run one, dropped in run two" becomes a visible diff (approved but never sent) instead of a silent skip, and the metric you want changes from "did the run finish" to "did every approved recipient get exactly one effect".
On retention: a time TTL feels wrong for this. A provider replay window like Stripe's 24h is a clock, but "the same email next month is legitimate" is a business rule. Since the key already scopes to the event, the event id is the natural expiry — a new event id never collides, so the row can live as long as you keep the business record of the event.
One question: for a tool whose provider has no idempotency support and whose effect is irreversible, do you treat the ledger write as the only barrier, or do you require a provider-side receipt you can look up later before the row counts as done?
Your point is now an issue on the repo: github.com/apowerb/apowerb/issues/477
If you’d like to contribute, comment there or open a PR. The tech team reviews every contribution, and you’ll be credited by name. Thanks for pushing the design forward!
Very good idea. To your question: not the ledger alone. The ledger stops us from duplicating our own effects; it doesn’t prove the effect happened. For an irreversible tool with no provider idempotency, a row would go pending before the call, done only on a provider receipt we can look up later (a message id, a transaction id), and unknown if the process dies in between. Unknown is never retried automatically, it goes to a human. That’s the rule our runs already follow: an unrecorded tool history means “unknown”, never “nothing happened”, and the replay is refused. A provider with no receipt at all never reaches done on its own.
The plan row fits naturally: our email-campaign skill already stops for approval before sending, and the approved list is exactly the set to freeze.
This deserves to be in the RFC. Would you post it there? github.com/apowerb/apowerb/issues
Thanks in advance for your contribution, and for the exchange, it was a really nice one.
You nailed the hard part: the key answers "have I done this, for this reason, to this target." We run Service Bus duplicate detection on a few flows, and the dedup key always comes from the business identifier, never the payload, for exactly this reason. On the ledger TTL question, scoping retention to the window where replays actually happen feels right. A duplicate from six months ago is usually a different request.
Thanks for the comment. Yes, Service Bus is a good reference here, since its duplicate detection makes the same call: business ID as the key, and a bounded window rather than forever. One constraint on our side: the window has to cover how long a failed run stays replayable, since a replay can be triggered by a human well after the crash, not just by an automatic retry. How do you handle flows where the business ID alone isn’t enough, like two legitimate messages for the same order?
The rule that the key comes from stable intent and never from generated content is the part I would pin above every agent tool. The email campaign that dies at 40 of 100 shows why both obvious answers fail: refuse the replay and 60 people get nothing, force it and 40 get the email twice.
Max Kaya's idea of freezing the intended target set as plan rows before the first effect fits well with what you already have. Once a human approves the campaign, that approval is the stable intent: a fixed list of recipients, each keyed on the event id and the resolved record, with nothing the model wrote. A replay then only has to ask which rows are still pending.
We use the same idea on client builds outside agents too. Write down what done looks like for each target before anything irreversible runs, and recovery becomes a lookup instead of a judgment call. Will the revoked state be visible to the person approving a forced replay?
Yes, that’s the whole point of the state. Today a forced replay only shows what the 409 lists: the tools the run already called. With per-target rows, the person forcing it should see each approved recipient as pending, done, revoked or unknown, with who revoked it and when. Only the pending ones go out. Revoked and unknown stay visible, and nothing happens to them unless someone picks them explicitly.
“Recovery becomes a lookup instead of a judgment call” is a great way to put it. And approval is the right moment to freeze the list, as Max suggested.
The pending row upthread covers the crash case. The same insert also covers one nobody has raised: parallelism inside a single turn. The Messages API can return several tool_use blocks in one assistant response, and the docs leave execution order to the caller, with a note that tools with side effects or shared state may be better run sequentially. Run them with Promise.all and two calls carrying the same (event, tool, recipient) key can both read an empty ledger before either writes. A select, then send, leaves that window open for the length of the request. The fix is the one your webhook section already uses: a unique index that rejects the duplicate insert, moved off (subscription_id, resource_id) and onto the effect key, with the pending row written before the call. The loser of the insert skips instead of asking. So: does that recipient loop live inside one tool call, or does the model get one tool_use block per recipient? Inside one call the loop is yours to serialise. One block per recipient hands the ordering to the runtime, and a batch call instead hides the race behind a key covering many targets, where a partial failure gets hard to attribute.
One tool_use block per recipient. The skill tells the model to send one call per recipient and not to parallelize, but that’s an instruction in the prompt, not a guarantee. If the model emits several send calls in one response, the ordering stops being ours, which is exactly your race.
So the guarantee has to live in the database, not the prompt: the pending row inserted against a unique index on the effect key, before the call, and the loser of the insert skips. We’d keep one block per recipient on purpose: each key covers exactly one target, so a partial failure stays attributable.
You clearly know this problem better than most. Want to open an issue for it on the repo, and take the PR? The ledger table with the unique index on the effect key and the pending insert before the call would be a great first contribution. The tech team will review it, and you’ll be credited by name: github.com/apowerb/apowerb/issues
Anna's two emails are the cleanest demo of this I've seen, since a reworded subject on replay is all it takes to break a sha256 of the args. The part I'd poke at is the soft-delete fallback in bi_datasets. Once a deleted row holds the deterministic ID, every replay of that import gets a fresh random ID, so two replays still make two datasets. Does the ledger's done-then-undone state end up replacing that fallback?
Good catch, and you’re right. Once a soft-deleted row holds the deterministic ID, the fallback draws a uuid4, so two replays of that import make two datasets. Idempotency holds until the first deletion, not after.
And yes, that’s the revoked state’s job: the deletion is the explicit action that bumps the epoch, so every replay after it lands on the same next ID. This one doesn’t even need the ledger: soft delete already stamps the row, so the fallback can be uuid5(original ID + the deleted row’s deletion time) instead of uuid4, walking the chain if that one was deleted too. Deterministic per deletion, and it still never resurrects the old row.
Small, self-contained fix with a clear test: delete, replay twice, one dataset. Want to take it as a PR?
Great post — default-deny 409 on crashed runs is the honest choice most people skip.
On the crash-gap question: if the ledger row is written after the tool call executes, a crash between the two leaves an effect with no record. Writing the row as pending before invoking the tool closes that hole — the next run sees pending, refuses to auto-replay, and it becomes an explicit operator decision. It doesn't make retry semantics any less thorny, but it turns an invisible gap into an auditable one.
A two-phase intent ledger is the only setup that survived our crashes when an external API lacks native idempotency. We insert a pending row with the target coordinates before the socket opens. If the process dies mid-flight, the restart loop checks pending rows against the external service audit log before deciding whether to reissue.
On key composition, adding an explicit action slug to the tuple solves the multi-message problem cleanly: (event_id, target, 'quote') versus (event_id, target, 'receipt'). If the tuple only tracks event and recipient, the agent cannot execute legitimate two-stage notifications for the same trigger.
For branching drift on a restart, treating dropped targets as intentional works better than trying to reconcile against the aborted run. If an agent dies after 40 sends and the second pass decides only 35 of the remaining 60 recipients qualify, the ledger blocks duplicates on the first 40 and lets the new turn handle the rest.
The pending row before the socket opens is the part most people skip, agreed. One thing that bit people we’ve talked to: audit logs that lag. If the restart checks before the provider has written the send, it sees nothing and reissues. Which service were you checking against, and did you have to wait out a window before trusting a miss?
Stable intent is the right root. One refinement: the key needs the identity of what the intent acts on, not just the intent. "Send the renewal email for invoice 7 to customer 42" replays safely. "Send the renewal email" dedupes two different customers into one. Intent plus the business identifiers it touches, never the generated body.
Agreed, and that’s why the table puts the record id and the normalized recipient in the key. One catch: those identifiers have to be resolved, not typed by the model. If the model writes “invoice 7” from context and gets it wrong on replay, that’s generated content again and a new key. The tool should look the record up and key on what it actually found.
This was a really interesting read. The point about hashing the arguments falling apart with LLM-generated content makes a lot of sense—what looks like the “same action” to us can easily produce slightly different arguments on a replay.
I especially liked the idea of building idempotency around stable intent and real-world identity rather than the generated text. That feels like a much more reliable foundation for agent systems.
The discussion around pending, done, and revoked was also a great touch. These are the kinds of edge cases that probably don’t show up in demos, but become extremely important once agents are performing real actions. Great practical perspective!
Thanks! That’s exactly the gap we kept hitting: everything looks fine in a demo, and the edge cases only show up once the agent touches real systems. If you’re running agents with real side effects, I’d be curious which ones bit you first.
Thank you for this insightful piece!
I'd freeze the approved email body too, not just the recipient list. Otherwise the remaining 60 people could get a rewritten message nobody approved. Keep generated text out of the idempotency key, sure, but keep the exact approved text as the payload to send on retry.
Agreed: the key decides whether to send, the frozen payload decides what to send. A replay should send the approved template, never a rewritten one. Added to the plan-row design: github.com/apowerb/apowerb/issues/477
Thanks for adding it. A useful regression test would be to change the source template after the crash, then resume. The remaining recipients should still get the exact text approved before the first send. That would catch accidentally storing a template reference instead of the actual payload.
That’s the right test, it catches exactly the bug that would look fine in every happy path. Added to #477 with credit to you: github.com/apowerb/apowerb/issues/477
The plan rows come right after the ledger table from #466. If you’d like to take the test or the PR itself, subscribe to #477 and watch the repo: you’ll be notified as soon as it’s ready to pick up.
Thanks for the credit, Anis! Glad the discussion turned into a concrete test. I can't commit to taking the PR right now, but I appreciate the invitation.
The tension I keep circling back to is between "same key, same skip" and a deliberately superseded attempt. If the first send failed at the provider and the replay drafts a better email, key (event, tool, recipient) is identical — so the ledger either replays a stored failure or skips a send the agent actively wants to redo. I'd add an intent epoch to the key, bumped only when the agent explicitly supersedes a prior attempt: dedupe then collapses only exact retries, never deliberate revisions. For the crash-between-effect-and-ledger gap, reconciliation reads on startup — a cheap read-only provider query per pending key — have held up better in my experience than any write-time ordering scheme.
The epoch is the right tool, with two conditions. First, a known failure shouldn’t hold the key at all: dedupe only on done, while pending and unknown block, so a send the provider rejected outright leaves the key free and the retry goes through without an epoch. The epoch is for the case where the first attempt really landed and you want a second, better one. Second, the bump can’t come from the model, or the model controls dedupe again and generated content is back in the key. A human or an explicit rule supersedes, the model only proposes.
Agreed on reconciliation reads at startup, with one caveat from the audit-log case upthread: a miss right after the crash isn’t proof. Some providers record the effect late, so a pending key needs a settle window before a miss counts as “not sent”.
Both conditions sharpen the design, and the first one actually dissolves the case I was worried about: if a provider-rejected send never occupies the key, the epoch only ever fires for landed-then-superseded attempts — which is exactly the one case that needed disambiguation. The second condition is the deeper one. Letting the model bump its own epoch would hand dedupe back to generated content through the side door, the exact failure mode the whole design exists to prevent. So the epoch stops being a data field and becomes a capability boundary: bumps come from outside the model, the same place high-stakes approvals come from. The open question is what counts as 'outside' — a human click is cleanest, but a deterministic orchestrator policy (an explicit replace-intent registered before execution) could hold the line without a human in the loop. Either way: if the epoch can only be bumped by something that can't be prompt-injected, the key stays model-proof.
Agreed, Alex, and “capability boundary” is the right name for it. My test for “outside”: could any token the model produced change whether the bump happens? If yes, it’s inside. A deterministic policy passes as long as it’s registered before execution and its inputs come from the workflow definition or the trigger, never from tool arguments. A human click is just the simplest case of that rule.
You’ve thought this through further than most. Want to open an issue for the epoch on the repo? It would fit right next to the ledger work, and anything you write there gets credited by name: github.com/apowerb/apowerb/issues
Opened it — thank you for the invitation: github.com/apowerb/apowerb/issues/473
It sits next to #466 as a companion: epoch semantics keyed to the ledger states (failed frees the key, only done can be superseded), the capability-boundary rule with your token-influence test as the conformance check, and the settle window carried over per epoch. Open questions at the bottom — column vs key string is the one I'd most like the team's call on.
Thank you, Alex, this is great. Pairing it with #466 is exactly right, and turning the token-influence test into a conformance check makes the rule testable instead of a convention.
The team will weigh in on column vs key string on the issue. My lean is a column, with the unique index on (key, epoch): the base key stays stable, so you can look up every attempt for the same intent, and “only done can be superseded” becomes something the database can enforce. Welcome aboard ;) !
Same lesson from a payment flow rather than agents: the key has to come from something decided before any generation happens. In a checkout I fixed, two concurrent webhook deliveries could produce two reports, because the "already done?" check wasn't tied to the order created at checkout. Keying on that order record fixed it. Agent tools feel like the same shape: fix the key when the intent is fixed, then let the tool refuse duplicates. Curious how you handle a replay where the intent itself legitimately changed. New key, or a different effect?
Same shape exactly, and “fix the key when the intent is fixed” is a great way to put it. When the intent legitimately changes, it depends on why. If something new triggered it, it’s a new event, so a new key comes for free. If it’s the same intent deliberately redone, it stays the same key with an epoch bump, decided outside the model (a human or an explicit rule) so generated content never controls dedupe. Alex Chen opened the design for that here, if you want to weigh in: github.com/apowerb/apowerb/issues/473