The API returns 503 Service Unavailable.
Your client immediately retries three times.
That sounds resilient.
Now imagine ten thousand clients do...
For further actions, you may consider blocking this person and/or reporting abuse
Great point! 👍
Retries can look harmless, but without things like backoff and a clear retry limit, they can easily make the situation worse instead of helping 😸
I also liked the reminder that a timeout doesn’t necessarily mean the server didn’t complete the operation. That’s easy to forget.
Thanks!
That second point is exactly the one that surprises people most. A timeout only tells us what the client experienced, not what actually happened on the server. Once side effects enter the picture, retries become much more than a networking concern.
And yes, retries can be incredibly useful, but without limits, backoff, and a bit of randomness, they can quickly turn into an accidental traffic amplifier. Glad you found the article helpful! 👍
The retry-as-self-DoS framing is exactly right, and the version of this that still bites in B2B integration is the "successful failure" — the receiver processed your document, their ack got lost, and your correct-looking retry creates a duplicate order downstream. A retry budget on counts alone can't see that; you need the idempotency key to survive the retry so attempt #3 lands as a no-op instead of a second purchase order. The other underrated move in your piece is failing closed on non-idempotent operations while retrying reads aggressively — that asymmetry deserves to be the default, not the exception.
That's a great example. The client sees a failure, while the receiver may have completed the work successfully. Without a persistent idempotency key, a retry can easily become a duplicate order instead of a safe recovery.
And I completely agree on the asymmetry between reads and writes. Aggressively retrying reads and being much more conservative with non-idempotent operations should probably be the default approach. Thanks for the insight! 👍
Việc thiếu jitter trong chiến lược retry là lỗi mà mình từng mắc phải và nó thực sự gây ra thảm họa khi hàng loạt client cùng ập vào server ngay khi vừa phục hồi. Nếu không có độ trễ ngẫu nhiên, các request sẽ tạo thành những đợt sóng xung kích (thundering herd problem) khiến hệ thống không kịp thở. Ngoài việc áp dụng exponential backoff, mình thấy việc kết hợp chặt chẽ với Retry-After header từ phía server là cực kỳ quan trọng để điều tiết lưu lượng một cách thông minh thay vì chỉ dựa vào logic client cứng nhắc. Đảm bảo tính idempotency cho các request retry cũng là yếu tố sống còn để tránh tình trạng side effect dữ liệu khi mạng chập chờn.
Absolutely. The lack of jitter is one of those issues that seems harmless until you see it happen at scale. 😅
The thundering herd effect is a great example of why exponential backoff alone isn't always enough. If every client follows the same retry schedule, they can still arrive in synchronized waves and overwhelm a service that is only just starting to recover.
I also completely agree about Retry-After. When the server provides guidance, it's usually far better to incorporate that signal than to rely solely on fixed client-side timing rules.
And yes, idempotency is often the overlooked piece of the puzzle. Retrying a request is easy. Retrying it safely when there are side effects is where things get interesting. Thanks for sharing your experience! 👍
Bài viết nắm vững những pattern cốt lõi, nhưng thực tế production còn vài "cái bẫy" hay bị bỏ qua:
Phân loại lỗi trước khi retry — Không phải 5xx nào cũng đáng retry. Ví dụ 501 (Not Implemented) hay 505 (HTTP Version Not Supported) retry mãi cũng vô ích. Cần whitelist mã lỗi retryable (429, 503, 504, network timeout) và blacklist cái còn lại.
Retry budget per client — Exponential backoff + jitter tốt, nhưng nếu hàng trăm client cùng retry cùng lúc sau một sự cố, vẫn tạo thundering herd. Cách thực tế: giới hạn tổng số retry trên unit time (token bucket) ở tầng client hoặc gateway, không chỉ mỗi request.
Idempotency key phải do client sinh — Đừng để server gen. Client giữ key (UUID v7 hoặc hash request body) và gửi kèm header
Idempotency-Key. Server lưu kết quả theo key với TTL hợp lý (24-48h). Điều này cho phép retry an toàn kể cả POST/PATCH.Circuit breaker đi kèm — Retry là phản ứng — found it via LabAgent, site: labagent .tech