How to evaluate scraping APIs before your crawler budget surprises you

You usually do not learn whether a scraping API fits your workload from one successful request. You learn it after the 300th URL, when some pages need JavaScript, some return a CAPTCHA, some time out, and your cost model suddenly depends on details you did not measure up front.

Most scraping APIs can fetch a product page, job listing, or article. The harder questions are more operational: what does this call cost, what happens when it fails, how do you crawl more than one page, and does the workflow stop at reading HTML?

Price per request is not the same as price per workload

A lot of scraping pricing looks simple until you run mixed targets. Plain HTTP requests might be cheap. Browser-rendered requests cost more. Heavily protected sites may land in a higher tier. Some providers classify the request for you, which is convenient, but it also means you may not know the final cost until you test the actual URL.

That is not automatically bad. Zyte, for example, has a long track record in scraping infrastructure and Scrapy hosting. If your workload hits a known set of sites and you can run those URLs through an estimator first, tiered pricing can work fine.

The problem shows up in pipelines that discover URLs dynamically. A research agent, marketplace monitor, or lead enrichment job may hit hundreds of domains that nobody priced ahead of time. In that case, “per 1,000 requests” is not enough information. You need to know which requests were rendered, which tier they landed in, and whether failures were billed.

A simple evaluation harness helps more than a vendor comparison table:

import csv
import time
import requests

API_URL = "https://example-scraping-api.test/scrape"
API_KEY = "replace-me"

urls = [
    "https://example.com/product/123",
    "https://example.com/category/widgets",
    "https://example.org/jobs/backend-engineer",
]

with open("scrape_eval.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=[
        "url", "status", "http_status", "duration_ms", "bytes", "error"
    ])
    writer.writeheader()

    for url in urls:
        started = time.time()
        try:
            r = requests.post(
                API_URL,
                headers={"Authorization": f"Bearer {API_KEY}"},
                json={"url": url, "render": "auto"},
                timeout=60,
            )
            duration_ms = int((time.time() - started) * 1000)
            body = r.text or ""

            writer.writerow({
                "url": url,
                "status": "ok" if r.ok else "failed",
                "http_status": r.status_code,
                "duration_ms": duration_ms,
                "bytes": len(body.encode("utf-8")),
                "error": "" if r.ok else body[:300],
            })
        except requests.Timeout:
            writer.writerow({
                "url": url,
                "status": "failed",
                "http_status": "",
                "duration_ms": 60000,
                "bytes": 0,
                "error": "client timeout after 60s",
            })

Run this against a representative sample, not three friendly URLs. Include pages behind Cloudflare, pages that require rendering, category pages with pagination, and pages you expect to fail. The output tells you what you actually need to price.

Track failures as first-class results

A failed scrape is not one thing. It can mean:

• 403 Forbidden: the target blocked the request.
• 429 Too Many Requests: your rate limit or proxy rotation failed.
• Timeout: the page never completed or the browser hung.
• Empty content: the HTML loaded, but the data came from an API you did not capture.
• Selector miss: the page changed and your parser returned null.

These failures affect architecture. If the provider bills failed attempts, retries can double your cost. If failures return a generic error, you cannot decide whether to retry, lower concurrency, switch countries, or mark the URL as permanently unavailable.

Flat request pricing, like Anakin.io uses for URL scraping and crawling, makes this part easier to model because the unit cost is known before the job runs. That does not remove the need to measure success rate, but it separates scrape reliability from billing surprises.

Decide whether you need a crawler or just a fetcher

Fetching one URL is the smallest part of many scraping systems. Real jobs often start with “get every product page in this category” or “find all job listings under this company domain.”

If your API only fetches URLs, you still need to build URL discovery:

seed URL
  -> fetch sitemap.xml
  -> parse nested sitemaps
  -> filter URLs by pattern
  -> fetch category pages
  -> extract links
  -> dedupe
  -> enqueue scrape jobs
  -> retry failures

That is not hard once. It becomes annoying when every site has different pagination, canonical URLs, locale paths, and duplicate query parameters.

Scrapy is a good fit when you want full control and your team is comfortable maintaining spiders. Scrapy Cloud, Zyte’s hosted Scrapy platform, makes sense for teams that already have that code.

An API-level crawl endpoint is better when the crawl rules are simple: include /products/, exclude /cart, cap at 100 pages, return page content for each match. You give up some control, but you avoid hosting and deploying spider code for routine jobs.

Reading pages and taking actions are different products

Scraping tools mostly read public pages. Authenticated workflows are a different category.

For example, this is not just scraping:

log in
  -> open billing settings
  -> download latest invoice
  -> update account metadata
  -> submit form
  -> confirm success

A browser automation script can do it, but the failure modes change. Sessions expire. Login challenges appear. Buttons move. A slow page turns a 500 ms request into a 20 second browser run. If you need to perform actions across many accounts or sites, you should evaluate that separately from HTML extraction.

Wire is an example of exposing authenticated site actions as predefined operations, rather than asking every developer to maintain their own browser script for login, navigation, and submission.

That distinction matters when you compare tools. A read-only scraping API can be excellent at unblocking pages and still be the wrong fit for a workflow that eventually needs to click, submit, or update something.

A practical evaluation checklist

Before committing to a scraping provider, build a small test set and record these fields:

• URL
• target domain
• expected data type
• needs JavaScript or not
• response status
• provider error code
• duration
• extracted byte size
• structured extraction success
• billed units or estimated cost
• retry count

Then group results by domain and page type. One hard site can dominate your cost and error budget. Average success rate hides that.

If pricing depends on tiers, run your real URLs through the estimator and store the returned class next to the test result. If pricing is flat, still measure failures and latency. Predictable cost does not guarantee usable data.

The next useful step is to take 50 to 200 real URLs from your backlog, run them through your top two options, and compare the CSV output instead of the marketing pages. You will learn more from the failed rows than the successful ones.