A retry is not a cheap second call. Every attempt sends the whole conversation again, so a retry costs the first call again, history included. Retry only what can change on the next try, like a 429 or a timeout.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Paul. That's an important cost consideration, especially when a retry resends a large context.
I'd separate two questions: is another attempt worth the cost, and is it safe to replay the operation?
For a side-effecting tool call, even a timeout can mean the receiver completed the action but the acknowledgment was lost. So I'd check the operation ID, idempotency record, or receiver-side evidence before replaying it, while also respecting the retry budget and current authorization.
How are you approaching that distinction in practice?
The 404 case is a good reminder that a status code describes the response, not what happened to the request. MDN's page for 404 only says the server "cannot find the requested resource", and it can't tell you whether anything ran before that.
One small addition to the "retry only what can change" point: 429 and 503 are the two where MDN's pages both mention a Retry-After header, so an agent can wait the stated time instead of using a fixed backoff. 400 and 403 are documented as "repeating the request without modification will fail with the same error", so those are safe to treat as never-retry without changes.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks for the thoughtful addition. I agree, especially with your point that a 404 response doesn't necessarily prove that no side effects occurred.
I think this suggests two separate questions an agent should answer before retrying:
Can the failure condition change? Respect Retry-After for 429/503 when available, and avoid repeating unchanged requests after 400/403.
Is repeating the operation safe? For state-changing operations, verify the outcome or use an idempotency mechanism before retrying.
That second distinction becomes particularly important for autonomous agents executing real workflows, where a duplicate action may have consequences beyond another failed HTTP request.
I'll incorporate this distinction into the article. Thanks for making the retry decision model more precise.
Reliable AI automation depends on handling failure scenarios just as carefully as successful executions. A retry mechanism that ignores whether the previous action actually took effect can create duplicate records, messages, or transactions. At IT Path Solutions, we consider execution verification, idempotency, and controlled retries important design principles when building agentic AI workflows for real-world applications.
This highlights why fault handling needs to be part of an AI agent's architecture from the beginning. In multi-step workflows, one incorrectly repeated action can affect everything that follows. At IT Path Solutions, we see execution tracking, idempotency, and outcome verification as essential building blocks when developing reliable agentic AI systems that interact with external APIs and business applications.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Glen. That's exactly the failure mode we wanted to surface.
In a small synthetic test, the receiver logged a 50-byte request body and deliberately returned HTTP 404. The caller saw an error, but we had evidence of receipt—not proof that the downstream business action had committed.
We're exploring how to preserve one logical operation identity across transport retries, keep the outcome "unknown" until reconciliation, and track confirmed steps separately in multi-step workflows.
For cases where neither a durable idempotency record nor a reliable readback endpoint is available, what recovery policy have you found workable in practice?
@suirindo My rule: a replay is only safe when the receiver can recognise an operation it already did. Without an operation ID or a check on its side, a timeout can't tell 'never ran' from 'ran, reply lost'.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Paul. I agree that a timeout alone cannot tell us whether the operation never ran or its reply was lost.
For side-effecting operations, a stable operation ID and receiver-side deduplication or authoritative readback are essential.
One nuance we're exploring is the consistency between an idempotency record and the actual business effect: recognizing an ID alone may still leave the outcome unresolved.
We'd preserve the same logical operation across transport attempts and reconcile an unknown outcome before considering another side-effecting call.
How do you handle upstream APIs that provide neither idempotency keys nor a reliable status endpoint?
The table separating 'never dispatched' from 'receiver recorded the body, then errored' is the useful part. A test I'd add is a fake receiver that records each request and then returns a 404, a 500, or nothing at all, with an assertion that the agent never replays the call before checking what the receiver recorded. Run the missing-response case at two different delays, because a retry that is safe after a fast failure can still double-apply when the receipt arrives late. Passing means the agent's next step is to reconcile or stop, and the replay count stays at zero.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Srinivas. That's a very concrete and useful test case.
I agree that a missing HTTP response must be treated as an unknown execution outcome, especially when the receiver may have already committed the operation.
Testing both immediate failures and delayed receipts is important. I'd make the key invariant explicit: zero replays while the execution outcome remains unresolved.
The agent should reconcile against receiver-side evidence, or stop and surface the uncertainty if reconciliation is inconclusive.
Correlation IDs and idempotency keys could also help make this behavior verifiable.
I appreciate the suggestion. This is exactly the kind of failure scenario worth turning into a regression test.
The constructed 404 is a good teaching case because it breaks the assumption most people carry: error means nothing happened. The receiver had the body, and the caller only saw a failure.
Junfeng and you landed on the detail that matters in practice. An idempotency key needs a state, not just presence, so "pending" past its timeout triggers a status lookup on the receiver rather than another send.
Kartik's point is the one I see missed most often. In many agent setups the retry is not in any branch the developer wrote. The model reads "404" as text and calls the tool again on the next turn. For sends, tickets and payments, that decision belongs in the harness, before the model sees the error.
On client agent builds at Ascendra Ventures we list every side-effecting tool up front and agree for each one how an unknown outcome gets resolved, because that is cheaper to settle in scope than after a duplicate invoice. For receivers with no status endpoint at all, do you lean toward blocking the action or asking a person?
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
I'd combine both: block automatic retries and escalate to a human for reconciliation. Human approval alone can't resolve an unknown external effect.
For payments or other high-impact actions, I'd require a reliable reconciliation or idempotency mechanism before enabling autonomous execution. The harness should enforce this, regardless of what the model decides next.
I'd also keep the same operation ID across retries and check the receiver's state before sending anything again.
A timeout only tells you that the caller didn't get a response. It doesn't tell you whether the side effect happened. And when the retry happens later, permissions and the target's state may have changed, so those need to be checked again too.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Vlad. Rechecking the target’s current state is an important addition. Even when duplicate execution can be prevented, a changed target or expired permission can make the original action inappropriate.
I’d keep the logical operation ID stable and record each attempt separately. Before another dispatch, the harness should check the previous outcome, current authorization, and the conditions that made the action valid.
That gives us two distinct checks: “Is another attempt safe?” and “Is this action still authorized and appropriate now?” An unresolved outcome should keep automatic replay blocked until there is sufficient evidence to proceed safely.
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
The three-way split (never dispatched / body arrived / unknown) is the useful bit. Most agent retry loops I see collapse all of those into "error → try again," which is how you double-charge or double-post when the receiver already took the body and then returned 500. One practical habit on the "unknown" row: park the work in a reconcile queue with the original idempotency key instead of replaying from the agent loop, so a human or a sweeper can ask the receiver what happened before a second dispatch.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks — this is exactly the operational distinction I was hoping to explore.
Moving UNKNOWN into a reconciliation queue makes the no-replay rule enforceable by the host, rather than dependent on the agent's next decision.
I'd preserve the logical operation ID, original idempotency key, request-body digest, attempt IDs, and authority snapshot in the queued record.
The sweeper would then obtain receiver-side evidence of a confirmed effect, confirmed non-effect, or still-unknown outcome. As you point out, receiving the body alone doesn't establish that the business operation committed.
One edge case I'd like to test is idempotency-key expiry. If the receiver discards its key record before reconciliation finishes, another dispatch using the same key could still produce a duplicate effect.
Would you keep that operation on HOLD until an authoritative lookup or human resolution?
This would make a valuable fault-injection test: the receiver commits once, returns HTTP 500, the agent resumes, and the sweeper reconciles without a second dispatch.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. [email protected]
the lost response case is the one that hurts with paid data calls, you cannot tell if the rows were already billed. I started passing the previous run id as an exclusion so a retry only returns people it has not seen
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Thanks, Kevin. Using the previous run ID as an exclusion is a practical way to avoid receiving the same people twice.
One distinction I'd want to verify is whether that also prevents duplicate billing, or only filters the returned records. If the first request was billed but its response was lost, the exclusion might prevent duplicate results while leaving the original charge unresolved.
Does your provider expose a billing or usage record tied to each run ID?
I'd treat result deduplication and billing reconciliation as separate checks. Until the original charge can be verified, the billing outcome should remain UNKNOWN, and any further paid retry should require a fresh decision based on that uncertainty.
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
When I trace these, the retry decision often isn't in any branch I wrote. The 404 goes back to the model as text and the next turn just calls the tool again. For side-effecting tools I'd keep that decision in the harness, so an error on a send or a ticket create gets checked against the receiver before the model sees it at all.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
A model can decide to call a tool again after reading "404", but for side-effecting operations the safety gate needs to be enforced by the harness.
In our synthetic test, the receiver logged the POST body before returning 404—evidence of receipt, not evidence of a committed business effect.
I'd tie attempts to one stable operation ID, reconcile with receiver-side evidence where available, and expose an explicit effect_unknown / retry_blocked state to the model if the outcome cannot be established.
Any further dispatch should also re-check current authority.
When the receiver offers neither an idempotency key nor a reliable readback endpoint, do you block and escalate, or use another recovery path?
@suirindo without a key or a status endpoint, I would stop calling it a failure. The outcome is unknown, and it stays unknown until the effect itself can be read back.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Agreed. An HTTP error doesn't prove the operation failed. Without an idempotency key or status endpoint, the outcome stays unknown. For side-effecting actions, we should avoid blind retries until the actual effect can be verified.
For side-effecting calls the idempotency record is the only reliable evidence. A 500 or a timeout can't tell you whether the receiver committed, but a key it already stored can.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Good point, Junfeng. A 500 or timeout tells us very little about whether the receiver committed the operation.
One nuance I'd add: the presence of an idempotency key alone doesn't necessarily prove completion. The record needs to be durably linked to the actual effect and its outcome.
A key marked as "processing" or "unknown" still leaves us with a reconciliation problem.
That's why I'd distinguish the logical operation from individual retry attempts, and verify the receiver-side outcome before replaying a side-effecting call.
How would you handle a case where the key exists, but the final effect is still uncertain?
Then the key needs a state, not just presence: pending until the effect is confirmed. A key still pending past its timeout triggers a status lookup on the receiver instead of a blind replay.
Founder & CEO of Netsujo. Building trustworthy AI agent systems, with a focus on task boundaries, verification, and production reliability. Creator of Agent Role Contracts (ARC).
Exactly. The idempotency key should track the operation's state, not just its existence. After a timeout, check the receiver's status. If the outcome still can't be verified, keep it unknown and avoid replaying side effects.
We're a place where coders share, stay up-to-date and grow their careers.
A retry is not a cheap second call. Every attempt sends the whole conversation again, so a retry costs the first call again, history included. Retry only what can change on the next try, like a 429 or a timeout.
Thanks, Paul. That's an important cost consideration, especially when a retry resends a large context.
I'd separate two questions: is another attempt worth the cost, and is it safe to replay the operation?
For a side-effecting tool call, even a timeout can mean the receiver completed the action but the acknowledgment was lost. So I'd check the operation ID, idempotency record, or receiver-side evidence before replaying it, while also respecting the retry budget and current authorization.
How are you approaching that distinction in practice?
The 404 case is a good reminder that a status code describes the response, not what happened to the request. MDN's page for 404 only says the server "cannot find the requested resource", and it can't tell you whether anything ran before that.
One small addition to the "retry only what can change" point: 429 and 503 are the two where MDN's pages both mention a Retry-After header, so an agent can wait the stated time instead of using a fixed backoff. 400 and 403 are documented as "repeating the request without modification will fail with the same error", so those are safe to treat as never-retry without changes.
Thanks for the thoughtful addition. I agree, especially with your point that a 404 response doesn't necessarily prove that no side effects occurred.
I think this suggests two separate questions an agent should answer before retrying:
Retry-Afterfor 429/503 when available, and avoid repeating unchanged requests after 400/403.That second distinction becomes particularly important for autonomous agents executing real workflows, where a duplicate action may have consequences beyond another failed HTTP request.
I'll incorporate this distinction into the article. Thanks for making the retry decision model more precise.
Reliable AI automation depends on handling failure scenarios just as carefully as successful executions. A retry mechanism that ignores whether the previous action actually took effect can create duplicate records, messages, or transactions. At IT Path Solutions, we consider execution verification, idempotency, and controlled retries important design principles when building agentic AI workflows for real-world applications.
This highlights why fault handling needs to be part of an AI agent's architecture from the beginning. In multi-step workflows, one incorrectly repeated action can affect everything that follows. At IT Path Solutions, we see execution tracking, idempotency, and outcome verification as essential building blocks when developing reliable agentic AI systems that interact with external APIs and business applications.
Thanks, Glen. That's exactly the failure mode we wanted to surface.
In a small synthetic test, the receiver logged a 50-byte request body and deliberately returned HTTP 404. The caller saw an error, but we had evidence of receipt—not proof that the downstream business action had committed.
We're exploring how to preserve one logical operation identity across transport retries, keep the outcome "unknown" until reconciliation, and track confirmed steps separately in multi-step workflows.
For cases where neither a durable idempotency record nor a reliable readback endpoint is available, what recovery policy have you found workable in practice?
@suirindo My rule: a replay is only safe when the receiver can recognise an operation it already did. Without an operation ID or a check on its side, a timeout can't tell 'never ran' from 'ran, reply lost'.
Thanks, Paul. I agree that a timeout alone cannot tell us whether the operation never ran or its reply was lost.
For side-effecting operations, a stable operation ID and receiver-side deduplication or authoritative readback are essential.
One nuance we're exploring is the consistency between an idempotency record and the actual business effect: recognizing an ID alone may still leave the outcome unresolved.
We'd preserve the same logical operation across transport attempts and reconcile an unknown outcome before considering another side-effecting call.
How do you handle upstream APIs that provide neither idempotency keys nor a reliable status endpoint?
The table separating 'never dispatched' from 'receiver recorded the body, then errored' is the useful part. A test I'd add is a fake receiver that records each request and then returns a 404, a 500, or nothing at all, with an assertion that the agent never replays the call before checking what the receiver recorded. Run the missing-response case at two different delays, because a retry that is safe after a fast failure can still double-apply when the receipt arrives late. Passing means the agent's next step is to reconcile or stop, and the replay count stays at zero.
Thanks, Srinivas. That's a very concrete and useful test case.
I agree that a missing HTTP response must be treated as an unknown execution outcome, especially when the receiver may have already committed the operation.
Testing both immediate failures and delayed receipts is important. I'd make the key invariant explicit: zero replays while the execution outcome remains unresolved.
The agent should reconcile against receiver-side evidence, or stop and surface the uncertainty if reconciliation is inconclusive.
Correlation IDs and idempotency keys could also help make this behavior verifiable.
I appreciate the suggestion. This is exactly the kind of failure scenario worth turning into a regression test.
The constructed 404 is a good teaching case because it breaks the assumption most people carry: error means nothing happened. The receiver had the body, and the caller only saw a failure.
Junfeng and you landed on the detail that matters in practice. An idempotency key needs a state, not just presence, so "pending" past its timeout triggers a status lookup on the receiver rather than another send.
Kartik's point is the one I see missed most often. In many agent setups the retry is not in any branch the developer wrote. The model reads "404" as text and calls the tool again on the next turn. For sends, tickets and payments, that decision belongs in the harness, before the model sees the error.
On client agent builds at Ascendra Ventures we list every side-effecting tool up front and agree for each one how an unknown outcome gets resolved, because that is cheaper to settle in scope than after a duplicate invoice. For receivers with no status endpoint at all, do you lean toward blocking the action or asking a person?
I'd combine both: block automatic retries and escalate to a human for reconciliation. Human approval alone can't resolve an unknown external effect.
For payments or other high-impact actions, I'd require a reliable reconciliation or idempotency mechanism before enabling autonomous execution. The harness should enforce this, regardless of what the model decides next.
I'd also keep the same operation ID across retries and check the receiver's state before sending anything again.
A timeout only tells you that the caller didn't get a response. It doesn't tell you whether the side effect happened. And when the retry happens later, permissions and the target's state may have changed, so those need to be checked again too.
Thanks, Vlad. Rechecking the target’s current state is an important addition. Even when duplicate execution can be prevented, a changed target or expired permission can make the original action inappropriate.
I’d keep the logical operation ID stable and record each attempt separately. Before another dispatch, the harness should check the previous outcome, current authorization, and the conditions that made the action valid.
That gives us two distinct checks: “Is another attempt safe?” and “Is this action still authorized and appropriate now?” An unresolved outcome should keep automatic replay blocked until there is sufficient evidence to proceed safely.
The three-way split (never dispatched / body arrived / unknown) is the useful bit. Most agent retry loops I see collapse all of those into "error → try again," which is how you double-charge or double-post when the receiver already took the body and then returned 500. One practical habit on the "unknown" row: park the work in a reconcile queue with the original idempotency key instead of replaying from the agent loop, so a human or a sweeper can ask the receiver what happened before a second dispatch.
Thanks — this is exactly the operational distinction I was hoping to explore.
Moving UNKNOWN into a reconciliation queue makes the no-replay rule enforceable by the host, rather than dependent on the agent's next decision.
I'd preserve the logical operation ID, original idempotency key, request-body digest, attempt IDs, and authority snapshot in the queued record.
The sweeper would then obtain receiver-side evidence of a confirmed effect, confirmed non-effect, or still-unknown outcome. As you point out, receiving the body alone doesn't establish that the business operation committed.
One edge case I'd like to test is idempotency-key expiry. If the receiver discards its key record before reconciliation finishes, another dispatch using the same key could still produce a duplicate effect.
Would you keep that operation on HOLD until an authoritative lookup or human resolution?
This would make a valuable fault-injection test: the receiver commits once, returns HTTP 500, the agent resumes, and the sweeper reconciles without a second dispatch.
the lost response case is the one that hurts with paid data calls, you cannot tell if the rows were already billed. I started passing the previous run id as an exclusion so a retry only returns people it has not seen
Thanks, Kevin. Using the previous run ID as an exclusion is a practical way to avoid receiving the same people twice.
One distinction I'd want to verify is whether that also prevents duplicate billing, or only filters the returned records. If the first request was billed but its response was lost, the exclusion might prevent duplicate results while leaving the original charge unresolved.
Does your provider expose a billing or usage record tied to each run ID?
I'd treat result deduplication and billing reconciliation as separate checks. Until the original charge can be verified, the billing outcome should remain UNKNOWN, and any further paid retry should require a fresh decision based on that uncertainty.
When I trace these, the retry decision often isn't in any branch I wrote. The 404 goes back to the model as text and the next turn just calls the tool again. For side-effecting tools I'd keep that decision in the harness, so an error on a send or a ticket create gets checked against the receiver before the model sees it at all.
That's a useful distinction.
A model can decide to call a tool again after reading "404", but for side-effecting operations the safety gate needs to be enforced by the harness.
In our synthetic test, the receiver logged the POST body before returning 404—evidence of receipt, not evidence of a committed business effect.
I'd tie attempts to one stable operation ID, reconcile with receiver-side evidence where available, and expose an explicit
effect_unknown / retry_blockedstate to the model if the outcome cannot be established.Any further dispatch should also re-check current authority.
When the receiver offers neither an idempotency key nor a reliable readback endpoint, do you block and escalate, or use another recovery path?
@suirindo without a key or a status endpoint, I would stop calling it a failure. The outcome is unknown, and it stays unknown until the effect itself can be read back.
Agreed. An HTTP error doesn't prove the operation failed. Without an idempotency key or status endpoint, the outcome stays unknown. For side-effecting actions, we should avoid blind retries until the actual effect can be verified.
For side-effecting calls the idempotency record is the only reliable evidence. A 500 or a timeout can't tell you whether the receiver committed, but a key it already stored can.
Good point, Junfeng. A 500 or timeout tells us very little about whether the receiver committed the operation.
One nuance I'd add: the presence of an idempotency key alone doesn't necessarily prove completion. The record needs to be durably linked to the actual effect and its outcome.
A key marked as "processing" or "unknown" still leaves us with a reconciliation problem.
That's why I'd distinguish the logical operation from individual retry attempts, and verify the receiver-side outcome before replaying a side-effecting call.
How would you handle a case where the key exists, but the final effect is still uncertain?
Then the key needs a state, not just presence: pending until the effect is confirmed. A key still pending past its timeout triggers a status lookup on the receiver instead of a blind replay.
Exactly. The idempotency key should track the operation's state, not just its existence. After a timeout, check the receiver's status. If the outcome still can't be verified, keep it unknown and avoid replaying side effects.