Circuit Breakers for Scrapers: When to Stop a Run That Is Going Bad

A crawler that never stops on its own is a crawler that will eventually spend your whole IP reputation on a bad hour.

I run a daily collector over public channel preview pages (no login, just the web view). One Tuesday it woke up, got a wall of clean 200s with structurally empty bodies, and kept marching through the queue for forty minutes. Nothing threw. Nothing logged an error. It just politely walked off the cliff, one polite request at a time.

That run taught me the rule I now enforce everywhere: a fetch loop needs a trip condition, not just a try/except.

The pattern in ten lines

class Breaker:
    def __init__(self, threshold=3, cooldown=600):
        self.threshold = threshold
        self.fail = 0
        self.cooldown = cooldown
        self.opened_at = None

    def record(self, ok):
        self.fail = 0 if ok else self.fail + 1
        if self.fail >= self.threshold:
            self.opened_at = time.time()

    @property
    def open(self):
        return (
            self.opened_at is not None
            and time.time() - self.opened_at < self.cooldown
        )

Three consecutive failures: the breaker opens, the run aborts, and the next attempt is deferred by a cooldown. Closed → open → half-open is the same state machine a payment gateway uses, and it maps cleanly onto collection runs.

The trap: your definition of "failure"

The breaker above is useless if ok means "no exception." Twelve months of logs convinced me that a run is only healthy when all three of these are true:

1. Status is 2xx — the cheap check everyone has.
2. Body has content — non-empty, above a sane byte floor. A 200 with a 900-byte consent shell is a failure.
3. Expected structure is present — for me: at least one message element in the preview markup. If the page shape my parser was written for isn't in the response, the response failed, full stop.

Score all three per page, feed the score into the breaker, and the "empty 200" cliff becomes a trip on page three, not a forty-minute walk.

Cooldown, not cancel

The half-open step matters more than the trip. If the breaker opens and your scheduler retries five minutes later from datacenter space that's currently distrusted, you convert a soft block into a hard one. Cooldowns should be boringly long — ten minutes minimum, an hour if the source has a history. A collection run that yields nothing for an hour costs you nothing. A /24 that gets classified as abusive costs you the quarter.

What the trip should do

Three things, in order:

1. Stop the loop — not "log and continue." Continue is the bug.
2. Keep the partial — pages collected before the trip are valid data. Persist them; don't throw away ninety percent of a run because the last ten percent went bad.
3. Emit a distinguishable signal — TRIPPED must not look like 0 new records, and neither should look like success. Your log has three states: good, empty, bad. Collapse any two of them and you will be blindsided by the third.

My daily job trips a breaker roughly twice a month now. Every trip used to be an outage I discovered by hand. Every trip now is a line in a log file that says: the world changed, the pipeline noticed, the pipeline stopped. That's the entire job.

I run this collector daily and sell the sourced briefs it produces. The pay-what-you-want sample (every claim with its source permalink) is here: https://heyuhe.gumroad.com/l/ruldgm — and if you want the wheel this pipeline runs on (pip-installable, no API keys, plus the five recipes above as a file): https://heyuhe.gumroad.com/l/scxtul.