Most “quick scrapes” fail the same way: unclear fields, no row cap, and silent nulls after pagination.
Before you open Playwright (or hire someone), write this scope card for any public source:
- URL pattern — exact pages or API endpoints (include pagination / sitemap).
- Public-only check — no login wall, no CAPTCHA farm, robots.txt / ToS allow automated fetch.
- Fields (≤12) — name each column; drop “nice to have” extras.
- Row budget — e.g. ≤5,000 rows. Caps force honesty about coverage.
-
Dedup key — what makes two rows the same (
id,url, …). -
Delivery — CSV plus a one-page
schema.mdwith row count, null rates, and the source pattern.
Prefer official JSON/API when it exists (?$limit= on open portals often beats HTML parse). Flatten nested fields once; note what you dropped.
Update (Oct 2026): the OpsPacket Public Data Pull service mentioned in the original version of this post is retired and no longer for sale. The scope card above still stands on its own.
Top comments (0)