An RSS feed grew to 500 items and my crawler got flagged as broken 📰

I didn't change a line of code. The feed changed, and Google News had been saying no the whole time.

Hi! I'm Tim, a self-taught developer in Nairobi. I build small data tools that run as Apify Actors, serverless jobs that people pay for per result.

Apify runs every public Actor once a day with its default input. The run has to finish within 5 minutes and return at least one result. If it fails three days in a row, the Actor gets an "under maintenance" label on its Store page.

This week one of mine got that label: a news crawler that reads RSS feeds, fetches each new article, pulls out the clean text, and remembers what it has already returned so a scheduled run only gives you what's new.

Over the last 30 days, 24 of its runs passed and 3 timed out.

Let's find out why! 🔍

⏱️ Reproducing the daily test

The daily test uses the default input, which for this crawler is an empty form. With no sources given, it falls back to four built-in ones: a Google News search plus the TechCrunch, Electrek and Engadget feeds.

So I ran exactly that on my machine and timed it:

580 seconds. Nearly twice the limit, and the two big problems are right there in the table.

📜 Problem 1: the feed was 500 items long

Electrek runs on WordPress, where a feed shows the latest 10 items by default. This one lists 500, and my code never asked how long a feed was. It fetched every item:

for (const source of sources) {
    let items;
    try {
        items = await discover(source);
    } catch (err) {
        log.warning(`Discovery failed for ${source.name}`, { error: (err as Error).message });
        continue;
    }
    for (const item of items) {
        const article = await fetchAndParse(item.url);
        // ...dedupe, save, charge
    }
}

The crawler is polite: it waits at least a second between requests to the same site. That's the right thing to do, but it means 500 articles from one site take at least 500 seconds before anything else happens.

Dedupe didn't help either. It runs on the parsed article (canonical URL, title, publish time and a hash of the body), so even an article the crawler had already seen got fetched again before it could be skipped.

🚪 Problem 2: Google News says no

The Google News source produced 100 links and 0 articles. Here's the start of news.google.com/robots.txt:

User-agent: *
Disallow: /
Allow: /$
Allow: /?
Allow: /home$
Allow: /home?
Allow: /home/
Allow: /nwshp$
Allow: /topics/
Allow: /publications/
Allow: /stories/
...

Everything is disallowed except the homepage and a few HTML sections. The RSS search feed lives under /rss/search, and every article link in it is a news.google.com/rss/articles/... redirect. None of those paths are allowed.

My crawler checked robots.txt before fetching each article, and correctly refused all 100. But it never checked before fetching the feed itself. So it read a feed it wasn't allowed to read, then politely declined every link in it.

By those rules, the Google News source can't return a single article. It only ever cost time.

(Further down, the same file has a block that disallows everything for GPTBot, ClaudeBot, PerplexityBot and a few other AI crawlers by name.)

🐢 Problem 3, found while fixing: timeouts got three retries

While testing the fix, TechCrunch article pages kept timing out from my connection. The fetch helper retried transient failures three times with backoff, and it treated a timeout like any other transient error.

With a 15-second timeout, that's 4 attempts plus backoff: about a minute for one page. A site that's having a bad day could eat the whole 5 minutes on its own.

🔧 The fixes

1. Check only the newest N items per feed

A new maxArticlesPerSource input, default 20. Items are sorted newest first, then cut:

const maxPerSource = Number.isInteger(input.maxArticlesPerSource) && input.maxArticlesPerSource! > 0
    ? input.maxArticlesPerSource!
    : DEFAULT_MAX_ARTICLES_PER_SOURCE; // 20

const newestFirst = (a: { publishedAt: string | null }, b: { publishedAt: string | null }) =>
    (Date.parse(b.publishedAt ?? '') || 0) - (Date.parse(a.publishedAt ?? '') || 0);

if (items.length > maxPerSource) {
    items = [...items].sort(newestFirst).slice(0, maxPerSource);
}

For a monitor that runs on a schedule, the newest 20 is what you want anyway. Anything older was either returned by an earlier run or is old news.

2. Check robots.txt at the door too

Feeds now go through the same robots.txt check as articles, and Google News sources are skipped with a reason instead of failing quietly:

export async function discover(source: Source): Promise<DiscoveredItem[]> {
    if (source.type === 'google-news') {
        throw new Error("Skipped: Google News doesn't allow automated access to its RSS feeds (robots.txt). Add the publishers' own RSS feeds instead.");
    }
    const feedUrl = source.feedUrl;
    if (!(await isAllowedByRobots(feedUrl))) {
        throw new Error(`Skipped: the site's robots.txt doesn't allow automated access to ${feedUrl}`);
    }
    const feed = await fetchFeed(feedUrl);
    // ...
}

Google News also left the default sources. Most publishers have their own feed, often at /feed or /rss, and those are the ones to use.

3. One retry for a timeout, and a per-site limit

A page that timed out once gets one more try, not three:

} catch (err) {
    const timedOut = (err as Error).name === 'TimeoutError' || /timeout/i.test((err as Error).message);
    if (attempt === MAX_RETRIES || (timedOut && attempt >= 1)) throw err;
    await sleep(2 ** attempt * 500);
}

And after 2 failed articles from the same site, the rest of that site is skipped for this run:

for (const item of items) {
    const site = siteOf(item.url);
    if ((failuresBySite.get(site) ?? 0) >= MAX_FAILURES_PER_SITE) {
        skippedDisallowedOrFailed++;
        continue;
    }
    // fetch, and on failure: failuresBySite.set(site, failures + 1)
}

The articles aren't lost. They're still in the feed tomorrow, when the site may be answering again.

4. "Nothing new" is a result

There was one more trap waiting. The daily test needs at least one result, but this crawler's whole point is to return only new articles. A second run on the same day, or a quiet news day, could return nothing at all and look broken.

So a run with nothing new now saves one row saying so, and doesn't charge for it:

if (stored === 0) {
    await Actor.pushData({
        title: 'No new articles in this run (not charged)',
        note: skippedDuplicate > 0
            ? `All ${skippedDuplicate} articles found were already returned by an earlier run.`
            : 'The sources returned no articles that could be fetched and read.',
        discovered,
        skippedDuplicate,
        skippedDisallowedOrFailed,
        skippedSources,
        discoveredAt: new Date().toISOString(),
    });
}

If you're scheduling a monitor, an empty result and a broken run look the same from the outside. A row that says why tells them apart.

🧪 After

The repeat run was slower because TechCrunch was timing out from my connection. The per-site limit kicked in after two failures, and it still finished well inside the limit. On Apify's servers, TechCrunch answered fine.

🎯 What I learned

• Inputs you don't control will grow. A feed, an API page, a sitemap. If the loop has no cap, the run time doesn't either.
• Check robots.txt at the entry point, not just the pages. My crawler was careful about every article and careless about the feed that listed them.
• A timeout is not a hiccup. Retrying a 15-second timeout three times turns one slow page into a lost minute.
• "Nothing new" deserves a row. For anything that runs on a schedule, say why the result is empty.
• A daily health check is a free alarm. The code hadn't changed since August. The world around it did, and the check is what told me.

The crawler is back to finishing in under a minute, and the under-maintenance label should clear after the next daily test.