If Claude and Codex Can Scrape the Web, Why Do We Still Need Web Scraping Tools?
This question has been on my mind lately.
Tools like Claude and Codex can already browse websites, read pages, extract information, and work with web content.
So if AI agents can do all of that...
Why would we still need a separate web scraping tool? 🤔
Maybe the difference starts showing when you need more than a few pages.
Hundreds of URLs. Structured output. JavaScript-heavy pages. Sessions. Repeated jobs. Reliable data that another part of your application can actually use.
I’m currently working in this space on Spicrawl, so I’m genuinely curious how other people see it.
Would you let your AI agent handle the whole web-data process itself, or would you still give it a dedicated scraping tool/API?
Would love to hear how you’re handling this in your projects.
Top comments (11)
My take: an agent is great at deciding what to fetch and weak as the fetch layer at volume. Once you're at hundreds of URLs on a schedule, you want deterministic code handling retries, sessions, pagination and output validation, with the agent calling it as a tool through a documented input schema. The Apify / Web Scraping Club 2026 survey points the same way: 45.8% of respondents use AI in scraping, and the holdouts gave trust in results, cost and unreliable performance on some sites as their reasons. Does Spicrawl expose an output schema that agents can validate against?
Agree. The agent decides what to fetch. The fetching should be simple, predictable code.
To your question: yes. You can send a JSON Schema with a CSS selector for each field. Spicrawl converts values to the right type ("$1,234.56" becomes 1234.56) and checks the result. With
strict: true, a page that doesn't match fails instead of returning half-right data, and it costs 0 credits. Fields with no match are listed inempty_fields.Errors also include a
retryableflag and a wait time, so your code handles retries, not the model.One limit: it doesn't follow pagination. You send the URLs. For large lists, a batch takes up to 10,000 URLs per call.
Nice, failing at zero credits instead of returning half-right data is exactly right. And the retryable flag is a great touch. Thanks for the detailed answer!
Glad it helped, Dan! Spicrawl is free now, so feel free to try it. We'd love your feedback, good or bad.
Thực tế là Claude/Codex chỉ phù hợp cho ad-hoc exploration — vài trang, dữ liệu nhỏ, one-off. Khi scale lên (hàng nghìn trang, chạy định kỳ, cần monitoring, retry logic, handle anti-bot, parse cấu trúc nhất quán), LLM trở nên đắt đỏ, chậm, và không kiểm soát được output schema.
Hai điểm thường bị bỏ qua:
Tóm lại: LLM là research assistant, scraping tool là data engineering infrastructure. Hai món không thay thế lẫn được (site: labagent .tech)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.