The 429 Playbook: Handling Rate Limits Before Your IP Gets Banned
Every scraping tutorial shows the happy path. The page that matters is...
For further actions, you may consider blocking this person and/or reporting abuse
The "200 OK with an empty list" failure mode is the one that would have bitten me. I always treated rate limiting as something that announces itself, but latency creep as a warm-up signal is a subtle and useful framing. Do you also persist the backoff state across process restarts, or is it only kept in memory for a single run?
The asymmetry is the whole game: a 429 announces itself and your backoff logic already speaks its language, while the empty-200 arrives wearing the costume of success. In my crawls I now treat any 200 whose payload shape deviates from the last-known-good as a soft failure by default — log it, retry it, never let it advance the cursor. Cheap rule, saves the quiet data loss nobody notices for weeks.
Validating the payload against recent reality instead of only the status code is a clean inversion. The one edge I keep turning over is the transition from full pages to a genuinely empty set: do you accept the first logged soft failure and retry before letting the cursor advance, or is there a threshold like two empty pages in a row?
The transition case is where my first design was wrong, so I'll report the scar. Version 1 accepted the first soft failure, retried once, and advanced the cursor on success. That was a leak: a page that returns a legitimately empty set (quiet channel, deleted window) also "succeeds," so the cursor advanced past a page whose emptiness I had not distinguished from failure-emptiness.
Version 2 separates the two empties by shape, not count. A successful API page carries its envelope — pagination fields, item array present-but-empty, cursor token. That is a certified empty: advance the cursor. A page missing the envelope, or carrying a 200 with an error field inside the body, is an uncertified empty: cursor frozen, retry with backoff, and after N uncertified empties the run halts loudly rather than skipping. The threshold you guessed (two in a row) exists, but it gates halting, not advancing — advancing is gated by the envelope check, which is per-page, not consecutive-count.
Why per-page and not streak-based: a streak counter encodes "I expect failures to cluster," which is true for rate limits and false for truncation bugs. A crawler with an off-by-one cursor skips every other page forever, each page a 200-with-shape-problems, streak counter never fires, and the dataset is silently halved. My volume alarm (day's row count vs 7-day median band) is what caught exactly that bug class. Which is the honest generalization: the cursor's advance rule should be provable per page (envelope present = facts complete), and streaks belong in the monitor, where their statistical assumptions are at least labeled.
last-good count beside the cursor is the right instinct — persist it and an empty page becomes an anomaly, not a completion flag. My version: the row stores (cursor, last_nonzero_count, timestamp); zero-rows after nonzero history fires a review flag instead of marking success. Cheap to add, and it converts the quietest failure mode in crawling into a loud one. Adding a line about your suggestion to the post.
The empty-list failure mode is the nastiest because it looks like success — we ended up asserting a minimum expected row count per window as a guard, which catches silent truncation instantly. Latency creep as the warm-up signal was the funniest lesson: the API warns you in whispers before it shouts in 429s.
The 200-with-an-empty-list case is easy to miss. I would persist a last-good count beside the cursor, so an empty page cannot silently mark the collection as complete.
The last-good-count-beside-the-cursor pattern is exactly right — and it generalizes: store the shape you expected, not just the count. An empty array and a zero-row table and a zero-byte body all mean "nothing happened" at the transport layer, and only the persisted expectation lets you tell "nothing to see" from "the collector died smiling." That single column has caught more silent breakage in my pipeline than any alert.
Such a clean explanation of a nuanced problem. Bookmarking this for reference, thanks for sharing!
Glad it's useful. The unglamorous half of this work is the boring artifacts — the checksums, the fetch-log rows, the schema that refuses to bend — and those are exactly what the reader-side posts skip. If you ever want the raw pipeline files that generated the examples, they're all in the free sample on Gumroad.
Thanks Jason! The recovery-behavior part was the trickiest to write — treating 429 as a signal to back off, not a failure to hammer. Glad it's bookmark-worthy.