Your agent sends a request. The tool returns an error. The agent tries again.
That sounds like recovery. It can also repeat an action that already happened.
The retry decision should depend on evidence about the earlier attempt.
Three situations that should not share one retry rule
Here, dispatch means handing the request to the component that sends it. The receiver is the service on the other end.
| What you can establish | What it tells you | Next decision |
|---|---|---|
| The operation never reached dispatch, with reliable evidence covering the attempted path | No request was sent through that path | Recheck current authority, target and input before considering another attempt |
| The receiver recorded the body, then returned an error | The body arrived; the error does not erase that fact | Inspect what effect occurred; use an established reconciliation or idempotency mechanism |
| The response is missing and receiver evidence is incomplete | Whether an effect occurred remains unknown | Preserve that uncertainty; investigate before an automatic replay |
These are policy choices to evaluate, not a universal recipe. Receiving a body also does not prove that a business operation completed. A service may validate, enqueue or reject it later.
Likewise, “nothing was sent” does not make an expired approval valid again. Evidence about the earlier effect and authority for the next attempt answer different questions.
A deliberately constructed 404
In an ARC synthetic loopback experiment, a local receiver deliberately recorded a fixed 50-byte body and then returned HTTP 404.
The caller observed the error. A separate receiver observation showed that the body had arrived.
We included two controls: a normal 204 response with one recorded receipt, and a no-send case with no dispatch or receiver connection. Repeating the request after the first send was refused by a consumed grant inside the test host.
That refusal was limited to one running process. The receiver ran under the same OS account. This was a fault-injection lab, not a discovered production incident, independent attestation, or a demonstration of durable recovery or exactly-once delivery.
Stable Agent Role Contracts checks supplied declarations. It does not supply a general runtime enforcement layer or prove that a real service deduplicated an action. We have not established an advantage over an equally informed existing retry/readback control.
A useful review question
Look at one side-effecting tool in your own system: sending a message, creating a ticket, or submitting a job.
Find the branch that handles an error. Check whether it distinguishes a request that never left from one that may have completed. Also check whether a new attempt retains the logical operation identity your receiver actually recognizes.
For a broader worked example, this earlier article separates approval, unknown outcomes and settlement.
Which evidence does your retry rule actually check: the response code, an idempotency record, receiver-side readback, or something else?
Top comments (25)
A retry is not a cheap second call. Every attempt sends the whole conversation again, so a retry costs the first call again, history included. Retry only what can change on the next try, like a 429 or a timeout.
Thanks, Paul. That's an important cost consideration, especially when a retry resends a large context.
I'd separate two questions: is another attempt worth the cost, and is it safe to replay the operation?
For a side-effecting tool call, even a timeout can mean the receiver completed the action but the acknowledgment was lost. So I'd check the operation ID, idempotency record, or receiver-side evidence before replaying it, while also respecting the retry budget and current authorization.
How are you approaching that distinction in practice?
The 404 case is a good reminder that a status code describes the response, not what happened to the request. MDN's page for 404 only says the server "cannot find the requested resource", and it can't tell you whether anything ran before that.
One small addition to the "retry only what can change" point: 429 and 503 are the two where MDN's pages both mention a Retry-After header, so an agent can wait the stated time instead of using a fixed backoff. 400 and 403 are documented as "repeating the request without modification will fail with the same error", so those are safe to treat as never-retry without changes.
Thanks for the thoughtful addition. I agree, especially with your point that a 404 response doesn't necessarily prove that no side effects occurred.
I think this suggests two separate questions an agent should answer before retrying:
Retry-Afterfor 429/503 when available, and avoid repeating unchanged requests after 400/403.That second distinction becomes particularly important for autonomous agents executing real workflows, where a duplicate action may have consequences beyond another failed HTTP request.
I'll incorporate this distinction into the article. Thanks for making the retry decision model more precise.
Reliable AI automation depends on handling failure scenarios just as carefully as successful executions. A retry mechanism that ignores whether the previous action actually took effect can create duplicate records, messages, or transactions. At IT Path Solutions, we consider execution verification, idempotency, and controlled retries important design principles when building agentic AI workflows for real-world applications.
This highlights why fault handling needs to be part of an AI agent's architecture from the beginning. In multi-step workflows, one incorrectly repeated action can affect everything that follows. At IT Path Solutions, we see execution tracking, idempotency, and outcome verification as essential building blocks when developing reliable agentic AI systems that interact with external APIs and business applications.
Thanks, Glen. That's exactly the failure mode we wanted to surface.
In a small synthetic test, the receiver logged a 50-byte request body and deliberately returned HTTP 404. The caller saw an error, but we had evidence of receipt—not proof that the downstream business action had committed.
We're exploring how to preserve one logical operation identity across transport retries, keep the outcome "unknown" until reconciliation, and track confirmed steps separately in multi-step workflows.
For cases where neither a durable idempotency record nor a reliable readback endpoint is available, what recovery policy have you found workable in practice?
@suirindo My rule: a replay is only safe when the receiver can recognise an operation it already did. Without an operation ID or a check on its side, a timeout can't tell 'never ran' from 'ran, reply lost'.
Thanks, Paul. I agree that a timeout alone cannot tell us whether the operation never ran or its reply was lost.
For side-effecting operations, a stable operation ID and receiver-side deduplication or authoritative readback are essential.
One nuance we're exploring is the consistency between an idempotency record and the actual business effect: recognizing an ID alone may still leave the outcome unresolved.
We'd preserve the same logical operation across transport attempts and reconcile an unknown outcome before considering another side-effecting call.
How do you handle upstream APIs that provide neither idempotency keys nor a reliable status endpoint?
The table separating 'never dispatched' from 'receiver recorded the body, then errored' is the useful part. A test I'd add is a fake receiver that records each request and then returns a 404, a 500, or nothing at all, with an assertion that the agent never replays the call before checking what the receiver recorded. Run the missing-response case at two different delays, because a retry that is safe after a fast failure can still double-apply when the receipt arrives late. Passing means the agent's next step is to reconcile or stop, and the replay count stays at zero.
Thanks, Srinivas. That's a very concrete and useful test case.
I agree that a missing HTTP response must be treated as an unknown execution outcome, especially when the receiver may have already committed the operation.
Testing both immediate failures and delayed receipts is important. I'd make the key invariant explicit: zero replays while the execution outcome remains unresolved.
The agent should reconcile against receiver-side evidence, or stop and surface the uncertainty if reconciliation is inconclusive.
Correlation IDs and idempotency keys could also help make this behavior verifiable.
I appreciate the suggestion. This is exactly the kind of failure scenario worth turning into a regression test.
The constructed 404 is a good teaching case because it breaks the assumption most people carry: error means nothing happened. The receiver had the body, and the caller only saw a failure.
Junfeng and you landed on the detail that matters in practice. An idempotency key needs a state, not just presence, so "pending" past its timeout triggers a status lookup on the receiver rather than another send.
Kartik's point is the one I see missed most often. In many agent setups the retry is not in any branch the developer wrote. The model reads "404" as text and calls the tool again on the next turn. For sends, tickets and payments, that decision belongs in the harness, before the model sees the error.
On client agent builds at Ascendra Ventures we list every side-effecting tool up front and agree for each one how an unknown outcome gets resolved, because that is cheaper to settle in scope than after a duplicate invoice. For receivers with no status endpoint at all, do you lean toward blocking the action or asking a person?
I'd combine both: block automatic retries and escalate to a human for reconciliation. Human approval alone can't resolve an unknown external effect.
For payments or other high-impact actions, I'd require a reliable reconciliation or idempotency mechanism before enabling autonomous execution. The harness should enforce this, regardless of what the model decides next.
I'd also keep the same operation ID across retries and check the receiver's state before sending anything again.
A timeout only tells you that the caller didn't get a response. It doesn't tell you whether the side effect happened. And when the retry happens later, permissions and the target's state may have changed, so those need to be checked again too.
Thanks, Vlad. Rechecking the target’s current state is an important addition. Even when duplicate execution can be prevented, a changed target or expired permission can make the original action inappropriate.
I’d keep the logical operation ID stable and record each attempt separately. Before another dispatch, the harness should check the previous outcome, current authorization, and the conditions that made the action valid.
That gives us two distinct checks: “Is another attempt safe?” and “Is this action still authorized and appropriate now?” An unresolved outcome should keep automatic replay blocked until there is sufficient evidence to proceed safely.
the lost response case is the one that hurts with paid data calls, you cannot tell if the rows were already billed. I started passing the previous run id as an exclusion so a retry only returns people it has not seen
Thanks, Kevin. Using the previous run ID as an exclusion is a practical way to avoid receiving the same people twice.
One distinction I'd want to verify is whether that also prevents duplicate billing, or only filters the returned records. If the first request was billed but its response was lost, the exclusion might prevent duplicate results while leaving the original charge unresolved.
Does your provider expose a billing or usage record tied to each run ID?
I'd treat result deduplication and billing reconciliation as separate checks. Until the original charge can be verified, the billing outcome should remain UNKNOWN, and any further paid retry should require a fresh decision based on that uncertainty.
When I trace these, the retry decision often isn't in any branch I wrote. The 404 goes back to the model as text and the next turn just calls the tool again. For side-effecting tools I'd keep that decision in the harness, so an error on a send or a ticket create gets checked against the receiver before the model sees it at all.
That's a useful distinction.
A model can decide to call a tool again after reading "404", but for side-effecting operations the safety gate needs to be enforced by the harness.
In our synthetic test, the receiver logged the POST body before returning 404—evidence of receipt, not evidence of a committed business effect.
I'd tie attempts to one stable operation ID, reconcile with receiver-side evidence where available, and expose an explicit
effect_unknown / retry_blockedstate to the model if the outcome cannot be established.Any further dispatch should also re-check current authority.
When the receiver offers neither an idempotency key nor a reliable readback endpoint, do you block and escalate, or use another recovery path?