The 429 Playbook: Handling Rate Limits Before Your IP Gets Banned
Every scraping tutorial shows the happy path. The page that matters is the one you get after an hour of collecting: 429 Too Many Requests.
Here's the playbook I run, ordered by what each response should cost you.
1. Read the headers first, back off second
Most servers tell you what to do in the response:
-
Retry-After: the contract. If it says 30, sleep 30. Not 5, not "I'll retry in a loop for a second". -
X-RateLimit-Remaining/X-RateLimit-Reset: your budget and its refill time. WhenRemaininghits 2, you throttle before the 429, not after. - No headers (common on free-tier APIs): treat every 429 as a warning shot. The next escalation is usually a temporary IP ban, and those are invisible — you get a 403 with a "blocked" page, not a rate-limit message.
2. The three-stage ladder
1st 429 → exponential backoff: 2^n * base (cap at 60s), 2 retries
2nd 429 → drop concurrency to 1, switch to a paced schedule (e.g. 1 req / 5s)
3rd 429 → stop. Persist the cursor, exit non-zero, let the scheduler retry later
The third stage is the one people skip. A tight retry loop that never gives up is how a free-tier API learns your IP address very well, very permanently. My collection jobs are resumable (cursor in a state file), so "stop and come back in an hour" is cheap — the pipeline picks up exactly where it died.
3. The failure mode nobody shows you
A 429 is not the only rate signal. Watch for:
- Silent partial responses: 200 OK with an empty list, where yesterday's identical call returned 50 items. Treat "200 + empty" as a soft 429 and re-verify against a known-good channel/endpoint before trusting the gap.
- Latency creep: responses slowing from 200 ms to 2 s is a rate limiter warming up. Throttle proactively.
-
Quality cliffs: the payload structure changes (
nullfields, truncated text) — some services degrade responses before they refuse them.
4. Budget math before you write code
Free tiers are rate-limited per IP, per key, per day. Do the arithmetic up front: 45 requests/min free tier × 60 min = 2,700 requests. At 200 items per page, a 20,000-message collection is 100 pages — fits easily on one channel, dies instantly at 30 channels. The playbook only works if your request budget covers the job; otherwise you're choosing between key rotation, sampling, or a paid tier today, not at 2 a.m. mid-collection.
The short version: 429s are a contract, not an obstacle. Read it, back off on schedule, persist your cursor, and know when stopping is the cheapest move in the pipeline.
Data from the same collection layer: Telegram OSINT bundle sample (PWYW) · GhostAction CI-security brief (PWYW) · weekly digest $15.
Top comments (11)
The "200 OK with an empty list" failure mode is the one that would have bitten me. I always treated rate limiting as something that announces itself, but latency creep as a warm-up signal is a subtle and useful framing. Do you also persist the backoff state across process restarts, or is it only kept in memory for a single run?
The asymmetry is the whole game: a 429 announces itself and your backoff logic already speaks its language, while the empty-200 arrives wearing the costume of success. In my crawls I now treat any 200 whose payload shape deviates from the last-known-good as a soft failure by default — log it, retry it, never let it advance the cursor. Cheap rule, saves the quiet data loss nobody notices for weeks.
Validating the payload against recent reality instead of only the status code is a clean inversion. The one edge I keep turning over is the transition from full pages to a genuinely empty set: do you accept the first logged soft failure and retry before letting the cursor advance, or is there a threshold like two empty pages in a row?
The transition case is where my first design was wrong, so I'll report the scar. Version 1 accepted the first soft failure, retried once, and advanced the cursor on success. That was a leak: a page that returns a legitimately empty set (quiet channel, deleted window) also "succeeds," so the cursor advanced past a page whose emptiness I had not distinguished from failure-emptiness.
Version 2 separates the two empties by shape, not count. A successful API page carries its envelope — pagination fields, item array present-but-empty, cursor token. That is a certified empty: advance the cursor. A page missing the envelope, or carrying a 200 with an error field inside the body, is an uncertified empty: cursor frozen, retry with backoff, and after N uncertified empties the run halts loudly rather than skipping. The threshold you guessed (two in a row) exists, but it gates halting, not advancing — advancing is gated by the envelope check, which is per-page, not consecutive-count.
Why per-page and not streak-based: a streak counter encodes "I expect failures to cluster," which is true for rate limits and false for truncation bugs. A crawler with an off-by-one cursor skips every other page forever, each page a 200-with-shape-problems, streak counter never fires, and the dataset is silently halved. My volume alarm (day's row count vs 7-day median band) is what caught exactly that bug class. Which is the honest generalization: the cursor's advance rule should be provable per page (envelope present = facts complete), and streaks belong in the monitor, where their statistical assumptions are at least labeled.
last-good count beside the cursor is the right instinct — persist it and an empty page becomes an anomaly, not a completion flag. My version: the row stores (cursor, last_nonzero_count, timestamp); zero-rows after nonzero history fires a review flag instead of marking success. Cheap to add, and it converts the quietest failure mode in crawling into a loud one. Adding a line about your suggestion to the post.
The empty-list failure mode is the nastiest because it looks like success — we ended up asserting a minimum expected row count per window as a guard, which catches silent truncation instantly. Latency creep as the warm-up signal was the funniest lesson: the API warns you in whispers before it shouts in 429s.
The 200-with-an-empty-list case is easy to miss. I would persist a last-good count beside the cursor, so an empty page cannot silently mark the collection as complete.
The last-good-count-beside-the-cursor pattern is exactly right — and it generalizes: store the shape you expected, not just the count. An empty array and a zero-row table and a zero-byte body all mean "nothing happened" at the transport layer, and only the persisted expectation lets you tell "nothing to see" from "the collector died smiling." That single column has caught more silent breakage in my pipeline than any alert.
Such a clean explanation of a nuanced problem. Bookmarking this for reference, thanks for sharing!
Glad it's useful. The unglamorous half of this work is the boring artifacts — the checksums, the fetch-log rows, the schema that refuses to bend — and those are exactly what the reader-side posts skip. If you ever want the raw pipeline files that generated the examples, they're all in the free sample on Gumroad.
Thanks Jason! The recovery-behavior part was the trickiest to write — treating 429 as a signal to back off, not a failure to hammer. Glad it's bookmark-worthy.