Timeouts and Retries: The Small Math That Prevents Big Outages

Most outages are not caused by the thing that failed. They are caused by everything that waited on it, and then retried it, all at once.

Timeouts and retries are the two smallest configs in your stack and the ones with the largest blast radius. The defaults are dangerous, the math is simple, and almost nobody does the math until after an incident review makes them. Here it is in advance.

The default timeout is forever

In Go, the standard HTTP client has no total timeout unless you set one:

client := &http.Client{} // a request can hang for as long as the other side takes

Other stacks have their own versions of this trap: a generous 60 or 120 seconds that nobody chose on purpose, multiplied by every worker you own. A dependency that stops responding does not produce errors in this setup. It produces silence, while your goroutines, threads, and connections pile up behind it. By the time anything times out, the queue is the outage.

The fix is not one magic number. The fix is that every outbound call carries a deadline, chosen on purpose, that fits inside the deadline of whoever called you:

ctx, cancel := context.WithTimeout(ctx, 800*time.Millisecond)
defer cancel()

req = req.WithContext(ctx)
resp, err := client.Do(req)

Passing the context down matters as much as setting it. When the deadline fires, the database query, the HTTP call, and the retry loop all stop together, because they share one clock.

Timeouts are a budget, not a setting

Picture a request with a two second budget at the edge. Your service needs 200 milliseconds for its own work, which leaves 1,800 for the services below. A downstream call that is allowed five seconds does not fit in that budget, no matter what its config file says. It will return its answer to a caller that stopped listening a long time ago, which means the work was pure waste, performed at the exact moment the system could least afford waste.

So budgets shrink as you descend. The edge allows 2,000 ms. Service A passes down 1,500. Service B gets 1,000, and its database queries get 700. Each layer keeps a slice for itself and hands the remainder down, always less than it received. If you cannot say your chain out loud in those terms, the numbers in your configs are fiction.

This is also why a timeout should be set from latency, not from hope. Take your p99 for the call, leave headroom for a bad moment, and cut it there. A call that normally answers in 300 ms and is allowed 30 seconds is not being given room to succeed. It is being given room to hold a connection hostage.

The retry multiplication problem

Retries feel like resilience. One extra attempt for one request is harmless. The trouble is that retries multiply across layers.

Say your client retries up to 3 attempts, your gateway retries 3, and your service retries 3 against the database. One user request becomes up to 27 database calls: 3 times 3 times 3. Now make the database the thing that is struggling. It gets slower, everything upstream retries harder, and the load on the failing component climbs to 27 times normal at the precise moment it can handle the least. This is the retry storm, and it is arithmetic, not bad luck.

The rule that prevents it is plain: retry at one layer only. Pick the layer closest to the failure, usually the direct caller of the flaky thing, and make every other layer pass errors through. If you truly need retries at two layers, cut the attempts so the total multiplication stays small, and write the worst case number down where reviewers can see it.

Backoff needs jitter

When retries do happen, the spacing matters. A fixed one second retry interval has a hidden property: every client that failed together retries together. A dependency blip fails a thousand requests in the same second, and one second later a thousand retries land in the same instant. The dependency falls over again, this time under a load spike your own clients manufactured.

Exponential backoff with jitter spreads that wave out:

func callWithRetry(ctx context.Context, fn func(context.Context) error) error {
    backoff := 100 * time.Millisecond
    var err error
    for attempt := 0; attempt < 3; attempt++ {
        if err = fn(ctx); err == nil {
            return nil
        }
        if !isRetryable(err) {
            return err
        }
        wait := time.Duration(rand.Int63n(int64(backoff)))
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(wait):
        }
        backoff *= 2
    }
    return err
}

Each client waits a random slice of its backoff window, so the retries arrive smeared across time instead of in formation. Note the two other details in that loop, because they are the ones that get skipped. It stops when the context deadline dies, so retries never outlive the budget. And it refuses to retry errors that retrying cannot fix.

Retry the right failures

Retryable means the failure smells transient and the operation is safe to repeat: a connection reset, a 503, a timeout on a read. Even then, the operation needs to be idempotent or protected by an idempotency key. A retried payment POST without one is not resilience. It is a double charge waiting for a slow network to introduce it.

Not retryable is most client errors. A 400 means you sent something invalid, and sending it three more times produces three more 400s plus load. A 401 will not become authorized by repetition. A 404 is an answer, not a failure. Retrying these wastes your budget and pollutes your error metrics with noise that hides real signals.

Database calls deserve the same honesty. A query that timed out because the table is enormous will time out again twice more, holding locks and connections each time. Retry the deadlock and the dropped connection. Do not retry the slow query. Fix the query.

Give retries a budget, not just a count

A per-request attempt limit still lets retries flood a system at scale. A thousand requests per second, each allowed two retries, is up to two thousand extra calls per second aimed at whatever is already failing. The count per request looks modest. The aggregate is a second incident stacked on the first.

Mature clients cap the aggregate instead. The rule is simple to state: retries may add no more than a fixed share of total traffic, say ten extra calls for every hundred original requests. Track attempts in a rolling counter, and once the budget is spent, calls fail fast until the share recovers. During a real outage this trims your retry volume automatically, at exactly the moment trimming matters, while in normal times the occasional transient failure still gets its second chance.

Pair that with a circuit breaker and the two controls divide the work cleanly. The breaker watches the dependency and stops calls entirely while it is down. The retry budget watches your own traffic and stops your clients from becoming the load problem while it recovers. One guards against their failure. The other guards against your enthusiasm.

The takeaway

None of this needs a framework. Every call gets a deadline. The deadlines nest inside one budget you can say out loud. Retries live at one layer, back off exponentially, and scatter with jitter. Idempotency decides what is safe to repeat, and error type decides what is worth repeating.

If you take one thing from this: find one outbound call in your code today, check its timeout and its retry count, and multiply that retry count by every layer above it. If the number surprises you, it will surprise you again during an outage, at a much worse hour.