A team I worked with tracked cost per task down to the token. They capped context. They set a budget alert. And then they added retries: 3 to the model client and never put it in the budget.
That retry path only fires when the model is slow or down — i.e. at peak load. And every retry replays the entire context. So the one part of the system nobody accounts for is the part that costs the most, exactly when you can afford it least.
The number that surprises people
Take a support agent for an ASEAN e-commerce shop. A hard ticket pulls a 50k-token context (order history, policy docs, prior turns) and emits ~1k tokens.
- One attempt ≈ 51k tokens.
- With
retries: 3and the task still failing, you pay for 4 attempts: 51k × 4 ≈ 204k tokens for a task you budgeted at 51k.
That's a 4× tax — and it isn't spread evenly. The tasks that fail are the expensive ones: long context, heavy tool chains, messy inputs. Retries concentrate exactly where cost is highest.
retry_policy:
naive:
attempts: 3
on_failure: "replay full context to same model"
hidden_cost_multiplier: 4 # 1 attempt + 3 full-context replays
shows_up_in_dashboard: false
cost_aware:
attempts: 3
on_failure: "route to next equivalent model"
carries_context: true
hidden_cost_multiplier: 1 # context travels, never replayed
shows_up_in_dashboard: true
Why the bill stays invisible
Most cost dashboards report average cost per successful request. A task that fails three times and finally succeeds doesn't appear as "4× cost" — it appears as one success with a latency blip. The retry is a hidden fourth axis of spend:
- Latency triples → someone notices.
- Cost triples → almost nobody measures it, because it's buried in "failed then retried" traffic.
So teams tune the happy path and ignore the one code path that runs under stress and multiplies the bill.
The fix is routing, not retrying
You don't make retries cheaper by retrying harder. You stop replaying context: when a model is unavailable, advance the task to the next equivalent model that can take the same handoff context. The context travels with the task; you don't re-pay for it as a "retry."
With 25+ models behind one OpenAI-compatible endpoint, "equivalent" stops being theoretical — there's a tier you can actually route to without rewriting prompts or adapters. For teams under data-residency rules (PDPA-aligned, SG-hosted), this also matters: re-running context on a different provider just to "retry" raises compliance questions, not just cost.
flowchart LR
subgraph Naive[Naive retry: context replayed]
N1[Task · 50k ctx] --> N2[Attempt 1: 50k in]
N2 -->|fail| N3[Replay 50k in]
N3 -->|fail| N4[Replay 50k in]
N4 -->|fail| N5[Replay 50k in]
N1 -.paid 4x.-.-> N5
end
subgraph Smart[Cost-aware: context travels]
S1[Task · 50k ctx] --> S2[Model A call]
S2 -->|unavailable| S3[Model B · same ctx]
S3 -->|unavailable| S4[Model C · same ctx]
S1 -.ctx paid once.-.-> S4
enddef step(task, model_registry):
# cost-aware: context is paid ONCE, then travels with the task
for model in model_registry.ordered_for(task.capability):
try:
return model.call(task.handoff_context) # no replay
except Unavailable:
continue # next equivalent, same context
raise NoCapableModel
# for contrast — the naive version re-sends the FULL context every attempt
def naive_retry(task, model, attempts=3):
for _ in range(attempts):
try:
return model.call(task.full_context) # full context RE-SENT each time
except (Timeout, Unavailable):
continue
raise RetryExhausted
The router doesn't retry the model. It advances the task to the next model that can take its context. Same uptime, a fraction of the bill.
The question I'll leave you with
Before you ship retries: 3: for the tasks that fail, how many tokens does the retry actually replay — and is that number in any dashboard you look at?
No tool to sell, no link to click. Just this: most LLM systems that survive a model outage are just retry loops with better logging — and the loop is the most expensive thing they do.
Top comments (28)
The 4× tax number is the one I'll quote. One addition from the ops side: the retry path also needs idempotency on tool-call results — replaying the full context against the same model isn't just expensive, it can double-apply a tool call if the first attempt actually succeeded server-side and only the response was lost. So it's the router you describe plus an idempotency check; either one alone still leaves a nasty failure mode.
Great addition — the silent-success case is exactly the one people miss when they treat "retry" as just re-running the loop. If the tool call already executed server-side but only the response was dropped, a naive context replay re-derives the same decision and fires it again. Double-apply.
One refinement I'd push: idempotency keys only help when the tool can take one. The cleaner split is to treat the tool layer as its own retry boundary, separate from the LLM context replay. You don't actually need to replay the whole context to retry a failed tool call — re-issue the call with the same idempotency key, and the context-replay loop (the expensive 4× part) only triggers when the model's decision failed, not the tool transport. Two different retry loops, two different cost profiles.
Which raises the messy case: tools with no native idempotency (a bare POST that won't accept a key). Do you wrap them behind an idempotency proxy, or just accept the double-apply risk on the low-stakes ones and only guard the destructive ones?
"Two different retry loops, two different cost profiles" is the cleanest way to frame this, and it's the part most agent frameworks get wrong.
They treat "tool call failed" and "model call failed" as the same thing, so they replay the whole context for both. But a tool call doesn't need the model's context — it just needs the same idempotency key.
Curious how you handle the "tool call succeeded but the result was lost" case. If the model never sees the result, it might decide to call the tool again with different arguments. Does the retry loop re-inject the original result, or does it let the model re-decide?
Good question — and it's the exact failure mode that breaks naive retries.
The fix is ownership: the retry controller, not the model, owns the tool result. On a silent-success (timeout, but the call executed server-side), you don't let the model re-decide. You key the tool call by signature — tool name + args hash — and cache the result; on retry you short-circuit and hand the model the cached result as if it came back clean. The model never gets a second pass at picking different arguments, because it's not in the loop for transport failures.
The trap is when retries live inside the model's own loop — then you can't stop the re-decision. So it's less "retry the tool" and more "the orchestrator replays the cached outcome." That only holds if the tool is idempotent-confirmed; otherwise you're replaying a side effect.
Which exposes the messier case: partial success. If the tool wrote something but you're unsure how much — a batch insert that committed half — the cached-result shortcut is wrong. You need either true idempotency or a fresh failure. How do you signal "ambiguous success" back to the orchestrator without forcing a full re-run?
"Ambiguous success" is the exact state that most retry systems don't have a word for.
You're right that the cached-result shortcut is wrong here, because replaying a partial write is worse than not replaying at all. The only honest answer is to treat "ambiguous success" as a first-class state, not as a failure or a success.
The orchestrator needs a way to say: "I don't know if this fully committed. Here's what I do know: the call was sent, the connection dropped, the tool reports partial completion." The model can then decide whether to retry, compensate, or escalate — but it decides with the actual state, not with a lie.
I'm not following the "context paid once" part.
model.call(task.handoff_context)still sends input to the next model. Is your gateway absorbing that cost, or is the handoff context smaller? Switching providers alone doesn't explain the drop from 4x to 1x.Fair challenge — "paid once" was sloppy framing on my part. Model B absolutely pays for what it receives; routing doesn't make tokens free. What it actually changes is three things: (1) the rate — cheap steps hit cheap models; (2) the handoff carries the slimmed context (results + deltas), not the full original replay; (3) a retry that doesn't re-attach the full original. The 4×→1× lives mostly in (2) and (3), not in "paying once." You're right that switching providers alone barely moves the number — the win needs the smaller handoff and a retry that doesn't re-bloat. How are you scoping what travels in the handoff today — full context, or a trimmed carry?
The observation that dashboards show cost per successful request is the uncomfortable one. Failed attempts disappear into the average, so the feature looks cheap right up until the invoice arrives.
SkepAgent's point deserves as much attention as the token math. A retry after a tool call that timed out but actually succeeded does not only cost tokens. It can apply the same change twice. So the order matters: make side effects idempotent first, then decide how many retries you can afford.
challan116's suggestion to budget retries as their own line item matches what we do on client builds. When a feature is scoped, it gets an expected cost per run and a separate allowance for failures, written down next to the acceptance criteria. If the failure line starts growing, that is a signal about the prompt or the model choice, and it shows up within days.
When you route to the next model and carry context forward, how do you stop the second model from repeating a side effect the first one already triggered?
The ordering point is the crux — idempotency has to be a precondition, not a retry setting. You can't retry your way out of a non-idempotent side effect; the retry just multiplies the damage.
On the cross-model case: the same "retry controller owns the tool result" boundary from earlier helps here too. If model B's attempt is a transport failure (not a decision failure), the cached result from model A's already-successful call short-circuits — so B never re-fires the side effect, it just receives the result as if it came back clean. The nasty case is partial: A triggered the side effect but the result was lost, and B is a different model that can't see A's cache. There you need a side-effect ledger keyed by task id, not by model — so any model in the chain can check "did this task's effect already land?" before acting.
Question: do you run that side-effect ledger as a separate service, or bake it into the orchestrator? Separate is cleaner to reason about but adds a hop on every call.
I'd lean toward keeping the ledger logically separate from the orchestrator, but not necessarily deploying it as a separate service from day one.
For state-changing financial operations, I'd initially persist the operation record, provider reference, and execution status alongside the transaction state in a durable database. The orchestrator can consult that record before deciding whether an operation is safe to execute again.
The important distinction for me is between a transport failure and an ambiguous operation outcome. If a provider may have completed the side effect but the response was lost, I'd reconcile the original operation before considering another execution.
I'm exploring this in Agentic Switcher, where the focus is deterministic orchestration, ownership fencing, and safe recovery across provider integrations. A separate ledger service could make sense later, but I'd first establish the consistency and recovery semantics.
One question I'd be interested in exploring is how you'd handle a side effect that succeeded externally but whose ledger update failed. That seems to be the hardest boundary to get right.
One distinction I think matters here is between a failed response and a failed operation. A timeout doesn't prove the side effect didn't happen.
For state-changing operations, I'd separate retrying the transport from retrying the operation itself. If the outcome is ambiguous, preserve the original operation reference and reconcile its status before deciding whether another execution is safe. Idempotency helps, but it doesn't automatically resolve partial completion or providers that don't support idempotency keys.
The audit trick that worked for us: log
cache_read_input_tokensnext to each attempt id, then join retries by task id — the replay tax falls out of the billing data as input-tokens-minus-cache-reads per attempt, no app instrumentation needed.One counterpoint to the routing fix: "equivalent" drifts. A fallback model with a different tool-calling format silently degrades the handoff context, so you trade a 4x bill for a quiet quality drop. Worth logging which tier actually served each attempt.
Does your handoff context carry tool definitions too? If model B doesn't expose the same tools, the task may need a re-plan rather than a rerun.
The audit trick is exactly right, and the part I like is that it needs zero app instrumentation — the replay tax just falls out of billing data once you join by task id.
On the format-drift counterpoint: yes, that's the silent killer. The tool schema isn't "data" you carry, it's part of the context the model reasons over. When model B speaks a different tool dialect, replaying the same tool_calls doesn't just degrade — it breaks, and the failure looks like a model problem, not a routing one. The split that's worked for us: the routing layer validates tool compatibility before handoff. If B can't express the same tools, you don't rerun — you either re-plan against B's toolset or you translate the schema. Rerunning the identical call is the worst of both.
Question for you: do you pin tool schemas per-model in a registry up front, or infer compatibility at call time? The up-front pinning catches the drift before it ships, but it rots the moment a provider changes a format.
The dashboard point is the one that generalizes furthest: any metric averaged per successful request will hide the retry tax, because the failures get folded into the cost of the eventual success. Retries also concentrate twice over — they fire at peak load, and they fire on the longest, messiest inputs — so the unbudgeted path is expensive in exactly the conditions where the system can least afford it. Budgeting retries as their own line item, capped by cost rather than by attempt count, is the version of this most teams could adopt without changing their architecture at all.
This is the most adoptable version of the whole argument — no architecture change, just accounting, which is why it'll actually ship in teams that won't touch their retry logic.
The "capped by cost, not attempt count" refinement is the key detail most people miss. An attempt-count cap lets a single huge context blow the budget while a cheap call gets throttled for no reason — the two failures have opposite shapes. Cost-capping flips it so the expensive replay is the thing that gets stopped.
One addition that makes it land sooner: surface the retry budget at the dashboard level, not just the invoice. You said it shows up "within days" when it's scoped next to acceptance criteria — it shows up within hours if the dashboard itself turns red when the retry line crosses its allowance. Visibility is what converts this from a post-mortem into a guardrail.
Question: do you budget retries per-route or per-feature? Per-route catches the one expensive endpoint; per-feature catches the feature that's quietly retry-heavy across routes.
"You don't make retries cheaper by retrying harder" is the core insight here, and it's the opposite of what most teams do.
The default response to "the model is down" is "retry with the same model, hope it comes back". But if the model is down, retrying it doesn't help — routing to a different model does.
Curious how you handle the "no equivalent model available" case. If all the models in the registry are down, does the task fail gracefully, or does it queue and wait?
"Retrying harder" is just re-paying the 4× tax with more determination — the insight holds, and it's the opposite of instinct.
On the no-equivalent case, it depends entirely on the task's latency contract, and the trap is treating both the same:
The mistake is a global "retry N times then give up" that applies the synchronous logic to async work (wasted waits) or the async logic to sync work (timeout blowups).
Question: do you split failure modes by task class (sync vs async, read vs write), or by a single global timeout? Class-based is more code but each path actually fits its caller.
"Class-based is more code but each path actually fits its caller" is the right tradeoff, and it's the one most teams avoid because it's harder to explain in a design doc.
The split isn't just sync vs async. It's also read vs write, and inside writes, idempotent vs non-idempotent. A sync read can fail fast and serve a cached result. A sync write needs to know whether it committed. An async read can queue. An async write can queue, but only if it's idempotent — otherwise you're queuing a duplicate.
Four classes, four different retry policies. It's more code, but each one is simpler than a global policy trying to handle all four.
The prefix cache tax is where this bites hardest. On long contexts, the first attempt hits warm KV cache and costs fractional input rates. The retry that fires during an outage or failover lands on a cold replica or a fallback model without cached state, paying full prompt ingestion prices four times in a row. A failure doesn't just multiply tokens; it drops you back to cold-start unit economics on the largest inputs in the pipeline.
This is the sharpest extension of the 4× thesis, because the tax isn't 4× the same price — it's 4× a higher unit price. You lose the warm cache, so each retry pays full ingestion on the largest context in the pipeline. The failure doesn't just multiply tokens, it multiplies the rate.
Two mitigations that have helped:
Question: have you measured the cache-hit-rate drop specifically on retry paths, or only on the aggregate? The aggregate hides it — the retry path can be near-zero hit rate while the overall number looks healthy, which is exactly why it slips past the dashboard.
Routing-the-task-not-the-model is the right instinct, and the token math is brutal once you see it. One thing I'd put before the retry-budget discussion: idempotency.
A timeout doesn't tell you the call failed — it tells you that you didn't hear back. If the side effect already ran (the payment went, the email sent, the row was written), "carry context forward and advance to the next model" can apply it twice. The cheapest retry policy in the world still double-charges someone if the thing it retried had a side effect.
So I'd gate it: reads retry freely; writes retry only behind an idempotency key, or re-grade the outcome (did the action actually happen?) before re-running. Otherwise the retry tax you measure in tokens comes back later as a reconciliation bill — same lesson as your dashboard point: the real cost is on the path you're not looking at.
Tokenade is my project, and a retry is not a cheap second try: it replays the whole context, so each attempt pays the full prefix again. Cost per successful task shows that, where cost per call hides it.
This is exactly the risk I designed around in a Node agent I run. Two things helped: a per-tool timeout (45s), so a hung data source fails once instead of stalling the whole turn, and falling back to a different model provider instead of retrying the same one. Retrying an identical long-context request against a provider that's already slow mostly buys you the same failure, at full context price.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.