My URL-to-Markdown converter chased a redirect chain into a consent wall — and learned to stop early

Had a real debugging session last week on the URL-to-Markdown fetch pipeline. A user fed it a link to a news article, and the converter kept returning an empty body with a 200 status. Looked fine on the surface — no error, just nothing.

Turned out the target URL was behind a soft redirect chain: article → tracking hop → CDN edge → cookie consent interstitial. Each hop returned 200. The final response was a fully-rendered GDPR consent page with zero article content. My fetcher followed all of them faithfully, scraped the consent banner, stripped the nav, and proudly returned a beautifully formatted Markdown document that said "We value your privacy" in seventeen languages.

The fix was unglamorous: cap redirect following at 3 hops, and bail early if the final response's Content-Type is HTML and the body's visible text is under some threshold. I also added a heuristic — if the page text contains phrases like "accept all cookies" or "consent" more than a couple times, flag it as a consent wall instead of returning it as content.

The interesting part: the managed alternatives (Firecrawl, Jina Reader, etc.) handle this by running a real browser and clicking the consent button. That works, but it costs 10-50x more per request and is slower. I chose the cheap path — detect and refuse — because for an agent feeding a RAG pipeline, a consent wall is worse than no content at least when it's silent. Better to return an explicit "blocked by consent" signal the agent can act on (retry with a different source, flag the URL for human review) than to silently ingest garbage into a vector store.

Lesson: in a fetch-and-convert pipeline, a 200 with empty meaningful text is a failure mode you have to hunt for explicitly. Status codes lie. Content-Type lies. The only signal left is "did this actually produce useful text?"

The converter now does that check, and I packaged the improved version at https://x402.freeq.one/tools/markdown.html — if you're feeding web pages into an agent context window, worth knowing what your fetcher is silently swallowing.