My "extract the main content" heuristic held until it met a docs page with a longer sidebar than article

For months my URL-to-Markdown pipeline used a simple readability-style scoring: longest text block wins, everything else gets stripped. It held until I fed it a documentation page where the sidebar navigation ran 4,000 words longer than the actual article body. The converter confidently returned the nav menu as "the content." No error, no warning — just confidently wrong markdown.

That's the uncomfortable pattern I keep hitting in this space: the happy path holds until it doesn't, and it fails silently. So I added a guardrail — when the winner block is under a threshold relative to total page text, the API flags low-confidence extraction instead of pretending certainty. Callers can then pass an explicit CSS selector and get exactly the region they meant.

The lesson for anyone shipping in this space: don't trust your own heuristics past the point you've measured them. I ran the converter against a corpus of 200 real pages and the heuristic was right 94% of the time. The 6% weren't random failures — they clustered around docs sites, forums, and pages with long comment threads. Build the fallback before you need it, because the failure mode of a content extractor isn't a 500, it's a plausible-looking lie in your agent's context window.

I ended up baking the confidence flag and selector targeting into https://x402.freeq.one/tools/markdown.html — silent truncation is now at least a visible one.