A team I worked with had a clean SLA: agent responses in under 8 seconds, 99.5% of the time. Then the frontier model they routed everything through started rate-limiting during peak hours. Their "failover" was three retries with exponential backoff.
The SLA held on paper. In practice, users got a 24-second response that sometimes failed anyway. The retries weren't failover — they were the same request, replayed, hoping the wall had moved.
The counterintuitive part
Retry and failover look identical in a dashboard ("request failed → tried again"). They are not the same operation.
A retry assumes the system was momentarily unavailable and the task is intact. That's true for a database connection. It's false for an agent mid-task: the moment you re-send the whole conversation, you've thrown away the in-flight state — the half-formed tool call, the retrieved doc, the branch the agent had already chosen. You pay for the first attempt's tokens and the replay's tokens. Latency triples. And if the model is down (not just slow), every retry hits the same wall.
So the failure mode isn't "generation failed." It's state continuity failed. The task had accumulated context, and retry discards it.
This is a contract problem, not a wrapper problem
You don't fix this by adding retries=3 to your client. You fix it by defining, per sub-task, what "equivalent" means — and making the fallback consume the same handoff context rather than "start over."
That requires three things most single-model setups don't have:
- An abstraction so the fallback speaks the same interface — if the backup model is OpenAI-compatible, swapping it in doesn't mean rewriting prompts or adapters. The task never learns a model went down.
- A ranked fallback registry, not a single backup — one standby model is a single point of failure with a different name. A registry orders equivalents by capability and cost.
- The routing layer carries state — the switch happens inside the task, not above it. The user sees one continuous attempt.
failover_policy:
primary: frontier-model
fallback_tier:
- model: mid-tier-instruct
equivalence: "handles same tool-calling schema"
carries_state: true
- model: small-instruct
equivalence: "degrades to summary, preserves context"
carries_state: true
on_primary_unavailable: route_to_first_capable # not: retry_primary
With a catalog of 25+ models behind one interface, "equivalent" stops being theoretical — there's a梯队 you can actually fail over to without dropping the task.
sequenceDiagram
participant T as Task (in-flight)
participant R as Router
participant P as Primary (frontier)
participant F as Fallback (mid-tier)
T->>R: needs next step
R->>P: call (carries handoff context)
P-->>R: 429 / timeout
R->>F: call (same handoff context)
F-->>T: continuation (state preserved)
Note over T,F: user sees ONE attempt, not a replayThe SLA only means something if the task survives
"99.9% availability" is meaningless if failover loses the task. A retry that returns after 3x the latency and a fresh start isn't a success — the user waited, paid, and got a do-over. The SLA number is green; the experience is red.
Real failover measures task completion, not request return. The fallback has to be able to pick up where the primary left off — which is exactly why the interface abstraction matters. Without it, you're not failing over, you're restarting.
def step(task, model_registry):
for model in model_registry.ordered_for(task.capability):
try:
return model.call(task.handoff_context) # state travels with the task
except Unavailable:
continue # next equivalent, same context
raise NoCapableModel
The router doesn't retry the model. It advances the task to the next model that can take its context.
The question I'll leave you with
Before you add a retry_on_failure wrapper: for the step you're protecting, what does "equivalent" mean — and does your fallback carry its state, or just rerun it?
No tool to sell, no link to click. Just this: most "resilient" LLM systems are retry loops with better logging. Resilience is a contract the fallback has to honor.
Top comments (4)
The replay-versus-continue distinction is the right one, and one failure sits underneath it: tool calls with side effects. If the primary dies after emitting a tool call (a write, a payment, a ticket) but before the result reaches the router, "carry the handoff context to the fallback" can make the fallback issue the same call again. The context says the step is still open, because nothing recorded it as done. So the handoff needs an idempotency key per tool call, or a log of calls started versus completed, and the fallback should see "started, outcome unknown" rather than "not started".
The retry arithmetic in your opening example also checks out as a lower bound: three attempts at roughly 8s each is already 24s before any backoff sleeps, so a p99 target of 8s cannot survive even one retry at peak, whatever the success rate.
When you tested the fallback tier, did you compare tool-call schema compliance between the frontier model and the mid-tier one? "Handles the same schema" is where I would expect the equivalence claim to break first.
The test I like for this: if the second attempt goes to the same place with the same request, it is a retry no matter what the dashboard calls it. Real failover changes something material — provider, region, or model — and it should also change what you log, because blending retry and failover metrics is how a degrading primary hides for weeks. Your SLA-on-paper point is the tell: latency budgets spent re-asking a rate-limited service are not resilience spending.
The distinction between discarding in-flight state and carrying the handoff context is the part that stuck. A retry that re-sends the whole conversation does look like failover in the metrics, and that is exactly why availability stayed green while the person on the other end got a do-over.
One edge that sits next to model failover: a tool call that timed out after send. Replaying the whole step under a new attempt can run the same side effect twice if the first call landed. Carrying state helps only if the pending tool outcome stays pending until the provider answers, instead of getting folded into "generation failed" and retried.
Curious whether your registry treats a mid-tool unknown as Unavailable (advance to the next model with the same intent still open), or parks the step until a reconciler closes it.
State continuity is the right name for it. The piece I would be careful with is the half formed tool call you mention. If the primary model already emitted a tool call and the tool ran, the fallback model must see that result, or it will plan the same action again and you get a duplicate side effect instead of a slow response.
So I think the handoff context needs two parts: what the model was thinking, which is nice to keep, and what already happened in the world, which is mandatory to keep.
In your registry, does the fallback get the executed tool results as facts it cannot redo, or does it see the same transcript and decide for itself?