DEV Community

whiteknightonhorse
whiteknightonhorse

Posted on

73 comments later: what we changed so a retrying AI agent doesn't pay twice

Last week I wrote about how we stop an AI agent from paying twice when it retries a paid API call. It got 309 views and 73 comments from 11 people, and most of the comments found a real gap.

This is the follow-up: what each gap was, what we changed, and what is still on the list. Everything below is live on our gateway unless I say otherwise.

The 409 looked final

Lin pointed out that many agent runtimes treat any 409 as terminal. Our payment_in_flight answer said "retry shortly" in prose and nothing else, so a runtime would give up and mint a fresh payment id on the next step.

The in-flight 409 now carries the fields a program can act on:

  • retry_after: 5 seconds
  • poll: shop.order.get, the call that shows the order state
  • order_id, when the order already exists

The agent guide says it plainly: payment_in_flight is not final. Wait retry_after seconds and send the same id again, or read the order.

payment_in_flight before and after: retry_after, poll, order_id

A crashed request kept its id hostage

Samii asked what happens when the first request dies outright. I checked, and it was worse than I thought. If our process died mid-request, the payment id stayed "in flight" until its 24 hours ran out. The reconcile job would settle the order from the chain, but the id row never learned about it. A retry could keep getting a 409 for an order that was already paid.

Now an in-flight row older than 180 seconds is resolved from the order itself. 180 is longer than the 120 second x402 payment timeout, so a first attempt that is still alive is never pre-empted. If the order is paid, the retry gets the stored order back with replayed: true. If it is still paying, the retry gets the 409 above. The row heals on the next request instead of waiting a day.

A stuck in-flight row is resolved from the order after 180 seconds

A conflict needed its own signal

Robert's point was that a conflict and an in-flight request mean different things to the client. One says the intent changed, the other says wait. They already had separate codes, but the conflict didn't say why. It now carries reason: payer or reason: request, and we alert on conflicts only. A plain repeat of the same request is normal and stays quiet.

The client has to write before it sends

Launchgatecheck named the assumption under all four server layers: the client can present the same id after a crash. That only holds if the agent stored the id before it sent anything.

The docs now say this as a rule. Store the intent first: the payment id, the quote and the recipient, plus the signed authorization itself. Then send. After a restart, send the same id again and never sign a new authorization for the same quote. The two stored things protect different things. The signed authorization bounds the money, because the single-use nonce fails on chain if it is replayed. The id recovers the work, because it is what gets the original result back. The guide also asks tool adapters to refuse to send a payment that has no stored intent.

Write the intent, then send: three steps for the agent

A retried lookup cost full price

This one came out of a long thread with Robert. Most of our tools are lookups. Over x402 a duplicated lookup was charged in full, even when the answer came from our cache, because the cache discount only existed on the prepaid balance.

Now, on x402, the same wallet asking the same cacheable lookup again within 120 seconds gets the answer without a second charge. The response carries X-APIbase-Charged: false and the receipt of the original paid call. It is deliberately narrow. It covers reads only, never writes, because two identical orders a minute apart can be exactly what the user wanted. Shop tools and the MPP rail are out of scope. The wallet is the session, so this assumes one wallet per agent, which is what our buyer docs already recommend.

Samii warned about volatile fields: a timestamp or cursor in the arguments makes every retry look new. A tool can now mark such fields so they are left out of the key. Nothing is left out by default, so the worst case of a missing mark is a missed match, not a wrong answer.

Duplicates on one rail had no refund record

The longest thread, with Kami, ended on an asymmetry. On Tempo, money moves before the work runs, so we had built a refund-owed record for failed or duplicate calls. On Base we settle only after the result is ready, so nobody had needed one. The edge case that broke this was a duplicate transfer landing after a failed settle. It would only have been found if someone looked.

There is now one refund-owed ledger for both rails. It records the two cases that matter here, a payment that arrived after the order was already final and a duplicate payment. The refund state shows up on the public receipt and in the order lookup, so the payer can see it.

Base x402 and Tempo MPP feed one refund-owed ledger

Still on the list

These are designed and queued, not live yet:

  • A targeted check after a failed settle on Base. We know exactly which authorization to look at, so instead of waiting for the periodic reconcile we read that nonce a few times over the next minutes. An inconclusive read stays unknown and falls back to the reconcile. It never becomes failed.
  • Automatic refunds from the ledger, paid in batches from a wallet set aside for it, so a payer doesn't have to care which rail they used.
  • Getting a paid result back by its receipt for 24 hours, signed by the paying wallet, for the agent that lost its runtime and comes back late.

What we decided not to do

A workflow-level id would merge two honest identical lookups in the same run, so the id stays per tool call. Writes never collapse on what the request looks like, only on an explicit key the agent minted. Per-session limits are easy to dodge because the server sees wallets, not sessions, so the hard boundary stays where Yamrail and I ended up: the wallet the agent pays from, with a cap on it.

Thanks to everyone who commented. Each change above started in that thread, so if you see a gap in this version too, I would like to hear it.

Top comments (17)

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

the write-the-intent-before-you-send section is the one that decides everything else, since every server-side fix here assumes the client can present the same id after a crash. making the 409 carry retry_after, a poll endpoint and order_id is the right shape, because it turns a terminal-looking status into a resumable one. the refund-owed ledger across both rails is the honest answer for the case where prevention fails anyway.

Collapse
 
whiteknightonhorse profile image
whiteknightonhorse •

Agreed, and that is the part we can't enforce from our side. If the agent dies before it writes the intent down, the retry arrives with a new id, and to the server it looks like a fresh purchase.

So the guide now tells agents to persist the id before the first send, and the ledger is there for the runs that skip it. Do you see clients actually doing that write in practice, or mostly holding the id in memory?

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

in practice it splits by how the client was built: teams with an existing outbox or audit log already make the durable write on every request, so adding the id there is one column and they converge fast. teams running a single in-memory script hold the id in memory and lose it on the first crash, which is exactly the cohort the ledger was built for.

Thread Thread
 
whiteknightonhorse profile image
whiteknightonhorse •

That split makes sense. The outbox teams get it almost for free, and the single-script runs are the ones a guide alone will not reach. Thanks for laying it out.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

thanks for coming back to it. that's the gap a guide alone keeps producing: a checklist protects the person who reads it right before shipping, and the single-script runs are exactly the ones nobody re-reads before. the teams that hold up are the ones where the idempotency key and the intent record are one line of shared client code, so the durable write happens whether or not anyone remembers the rule.

Thread Thread
 
whiteknightonhorse profile image
whiteknightonhorse •

Fair point, and it fits the rest of this thread. The rule holds when it sits in the client code people already run, not in a page they may or may not read before shipping. That one is going back to the team.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

that's the right way to close it out: the rule only survives shipping when it's enforced where the retry loop lives. if it makes it into the next version, worth a line in the changelog so anyone who read the guide picks up the fix too.

Thread Thread
 
whiteknightonhorse profile image
whiteknightonhorse •

Agreed. If it ships, I'll ask for a changelog line that links back to this thread, since most of the reasoning came out of it.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

a changelog line pointing back here is worth doing. threads like this tend to be the only place the reasoning survived, the release note captures the what and the thread keeps the why. anyone debugging the same retry path six months from now will thank you for the breadcrumb.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Tokenade sits between the agent and the model, so from that side a retry is a second full request: the context goes out again, and its input tokens count again, at the cache rate only while the prefix is still cached. An idempotency key covers the payment, not those tokens.

Collapse
 
whiteknightonhorse profile image
whiteknightonhorse •

Right, the key only stops the second charge for the tool call. The tokens an agent spends sending its context again sit on its side of the line, and the server never sees them.

The cheaper path is to check the order state before any retry and resend only when that check shows the first call failed. How often do you see agents resend the whole context where a short status check would do?

Collapse
 
challan116ux profile image
challan116-ux •

The follow-up is more valuable than the original post, because the failure modes only show up once other people describe how their retries differ. The principle I keep coming back to is that the payment itself needs a durable identity tied to the logical purchase, checked before execution — not a lock, not a timeout, but a record the retry can find and recognize as its own earlier success. An agent makes this harder than a classic worker because the retry may re-plan and arrive with slightly different arguments, so the identity has to live at the level of intent, not at the level of the exact request bytes.

Collapse
 
whiteknightonhorse profile image
whiteknightonhorse •

That is where we ended up too. The identity is the payment id the agent writes down with the quote and the recipient before it sends anything. A re-planned retry reuses that id and does not rebuild a key from the new arguments.

For the arguments, a tool can mark volatile fields like a timestamp so they stay out of the key. If the real arguments change under the same id, the agent gets a conflict with reason request, not a second charge. The case we still worry about is an agent that re-plans and loses the stored id. How do you handle that one?

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@whiteknightonhorse I don't have a count of how often agents resend blindly, so I won't put a number on it. What I can say: a status check only wins when the context is large, and that is exactly where a resend hurts most.

Collapse
 
whiteknightonhorse profile image
whiteknightonhorse •

That matches what we see. A small lookup is cheaper to resend than to ask about. On our side an identical repeat of a cacheable call from the same wallet within two minutes comes back from cache and is not charged, so the cheap case mostly takes care of itself.

The large context is where it hurts. Do you let the agent decide when to check status, or does the runtime force it once the payload crosses some size?

Collapse
 
yamrail profile image
Yamrail •

Appreciate the mention on the hard boundary!

In fact, intent_budget.py was born directly out of our conversation—I proposed cutting losses, and you made me realize the danger of double-paying on retries. I ended up implementing that exact lesson into our hackathon build this weekend and just opened up the repo:

github.com/Yamrail/Code_CLI
(specifically source/Code_CLI/native_runtime/payload/implementation/intent_budget.py)

Thanks again for the great architectural sparring!

Collapse
 
whiteknightonhorse profile image
whiteknightonhorse •

Thanks for sharing the repo, and good luck with the hackathon results. I read intent_budget.py. The part I like most is that a partial overlap in scope or lineage returns UNKNOWN_HOLD and not ALLOW. Treating "not an exact match" as safe is exactly where a double spend slips through.

What happens after a hold in your build? Does a person clear it, or does the agent get to retry with a narrower identity?