DEV Community

Kayvan Zahiri
Kayvan Zahiri

Posted on Edited on Originally published at dev.to

Before you build a scraper: a public-data CSV scope card that keeps one-offs honest

Most “quick scrapes” fail the same way: unclear fields, no row cap, and silent nulls after pagination.

Before you open Playwright (or hire someone), write this scope card for any public source:

  1. URL pattern — exact pages or API endpoints (include pagination / sitemap).
  2. Public-only check — no login wall, no CAPTCHA farm, robots.txt / ToS allow automated fetch.
  3. Fields (≤12) — name each column; drop “nice to have” extras.
  4. Row budget — e.g. ≤5,000 rows. Caps force honesty about coverage.
  5. Dedup key — what makes two rows the same (id, url, …).
  6. Delivery — CSV plus a one-page schema.md with row count, null rates, and the source pattern.

Prefer official JSON/API when it exists (?$limit= on open portals often beats HTML parse). Flatten nested fields once; note what you dropped.


Update (Oct 2026): the OpsPacket Public Data Pull service mentioned in the original version of this post is retired and no longer for sale. The scope card above still stands on its own.

Top comments (0)