Autonomous AI agents need clean web data, but micro-subscriptions and API key signups create friction. To solve this, I built and deployed x402 Web Scraper API—a pay-per-request web scraping service gated by HTTP 402 Payment Required micro-payments on Base.
Instead of managing accounts or monthly subscriptions, an AI agent requests scraped markdown directly, receives a 402 payment challenge for $0.005 USDC, settles on-chain via Base mainnet, and receives the structured data in a single flow.
🛠️ How It Works
Agent Requests Scrape: Sends a POST request to /api/scrape with a target URL.
Server Challenges: The server returns HTTP 402 Payment Required with payment parameters (amount, recipient address, asset address, and chain).
Agent Settles: The agent executes an on-chain transfer on Base Mainnet.
Data Returned: Upon payment verification, the server returns LLM-ready content.
🧪 Quick Start & Client Integration
You can integrate this with your agent using the official client library:
Bash
npm install x402-scraper-client
Here is a simple Node.js agent implementation intercepting the x402 payment flow:
import fetch from "node-fetch";
async function runAgent() {
const targetUrl = "https://news.ycombinator.com";
console.log(🤖 Agent requesting web scrape for: ${targetUrl});
const res = await fetch("https://x402-scraper-api-production-67a4.up.railway.app/api/scrape", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url: targetUrl })
});
if (res.status === 402) {
const paymentReq = await res.json();
console.log("💰 x402 Payment Gate Triggered!");
console.log(" Protocol:", paymentReq.protocol);
console.log(" Network:", paymentReq.network);
console.log(" Asset:", paymentReq.asset);
console.log(" Amount:", paymentReq.amount, "units ($" + paymentReq.price_usd + " USD)");
console.log(" Recipient Wallet:", paymentReq.recipient);
} else {
const data = await res.json();
console.log("✅ Scraped Data:", data);
}
}
runAgent();
🔗 Resources & Specs
OpenAPI Spec: https://x402-scraper-api-production-67a4.up.railway.app/openapi.json
AI Plugin Manifest: https://x402-scraper-api-production-67a4.up.railway.app/.well-known/ai-plugin.json
GitHub Repository: ihen404/x402-scraper-api
I'd love to hear feedback or see what agent tools you're building with x402! Feel free to leave a comment or drop a PR on GitHub.
Top comments (1)
This is a neat fit for 402 — scraping is exactly the domain where an API key and signup flow add more friction than the payment itself. I run scraping infrastructure (headless browser bridge, CDP-driven agents) and the account onboarding is consistently the most brittle part of consuming data APIs, not the scraping.
One failure mode I'd want to know how you handle: the agent settles the on-chain transfer, then the connection drops before your server returns the markdown. The agent has paid but holds nothing. Does the client get a receipt it can replay against
/api/scrape, or does the pattern assume you just pay again? At $0.005 nobody goes broke, but a retry loop on a flaky network is still a loop of on-chain transactions, and your verification needs to distinguish "paid once, deliver twice" from two distinct paid requests.Related: what does end-to-end verification latency look like? Base transfers confirm in seconds, but if your verifier waits for N confirmations or leans on a third-party indexer, per-request latency could dwarf the actual fetch. For agents polling in a loop that compounds fast.
Also curious about cache semantics: same URL requested by two different paying agents — refetch every time, or shared cache behind the payment wall? The economics of $0.005/request only work if the marginal cost per request is a fraction of a cent, which is mostly a scraping-infrastructure question, not a payment one.