I run a travel site that tells visitors what the cannabis laws are wherever they're going. This week I re-checked every US state against its official sources. The pages needed it. One said Nevada allows an ounce; the limit has been two and a half ounces since January 2024. One said Colorado allows an ounce; it's two, though you can only buy one at a time. Massachusetts doubled its limit in April 2026 and the page hadn't noticed.
Fifty-one jurisdictions, each with a possession limit, a penalty, a rule for visitors and a source, is too much to check by hand in a week. So four AI checkers did it, and the interesting part is not that they found the old errors. It's that each of them made new ones, and none of those reached the site.
The four checkers
A coding agent (Claude Code, on a home server) drafts every change: it reads the old page, searches for what changed, writes the new text and sends the questions it can't settle.
A browser agent gets those questions. It's a separate AI agent with its own real browser, and I talk to it through a Slack channel. Each request is numbered, lists the claims to confirm, and asks for the exact wording from the official page plus the URL. "NOT FOUND" is an accepted answer; guessing isn't.
A critic watches that channel. When the browser agent posts an answer, a second instance of Claude Code reads it in read-only mode and replies in the same thread within seconds: which claims are sourced, which aren't, which contradict what's already on the site. It can't change anything. Nothing is applied until I say "apply it".
A classifier grades the final output. Jev is a decision model: you give it text and a question with fixed answer options, and it picks one with a confidence score. It can't look anything up, which is the point. It only judges whether a conclusion follows from the text in front of it.
The browser sees what the server can't
The first surprise was physical. The coding agent runs on a server outside the US. Of the 39 source URLs it collected for the first half of the states, 17 wouldn't load from there: nine returned 403 (bot protection) and eight never answered at all. Those included Florida's statutes, Idaho's, Kentucky's, Nebraska's, Utah's and Wisconsin's.
The browser agent opens all of them, because it's a real browser on a US connection. So the work splits by what each side can reach, not by preference: the server checks every page it can load, and the browser agent quotes the rest.
Everyone was wrong somewhere
The browser agent said Tennessee banned THCA products from July 1, 2026. The state regulator's FAQ says something else. THCA has counted toward the 0.3% THC limit since January 1. July 1 was the date by which stores must buy only from licensed wholesalers. A real date, attached to the wrong rule. The coding agent caught it by checking the regulator's FAQ before applying the answer.
The browser agent also said a 2025 law opened Iowa's medical program to non-residents. The critic flagged that there was no source. The rulemaking notice turned up: it's a 2026 law, and in August the rules weren't final. The page now says to check with the state first.
The critic doubted that Maryland fines public smoking $50 the first time and $150 after that. Its "understanding of the law" was a flat $50. The statute says $50, then $150. The browser agent was right, and the critic's doubt cost one lookup.
The critic also warned that Maine's concentrate cap was 5 grams, because the original law said so. The state cannabis office's FAQ says 10 grams. Again the browser agent was right.
The coding agent labelled Washington DC "with a medical card": you need a card from home to buy there. The browser agent read the DC regulator's page and corrected it. Visitors don't need a home card at all; anyone 21 or over can register online as a temporary patient. My label wasn't just imprecise. It undersold the one thing a visitor wants to know.
The classifier graded all 51 "can visitors buy?" labels against the text of each state's rules, and disagreed with one. I had North Carolina as a flat "no". Jev said yes, at 0.83 confidence, because the text says a tribal store on Cherokee land sells to any adult 21 or over. Under state law the answer is no. On the tribal land the answer is yes. The label now says both, because a classifier with no world knowledge noticed that my "no" contradicted my own paragraph.
The pivot
I expected the checkers to stack: each one a stricter filter, so that whatever survived all four was right. That isn't what happened. The critic was strictest, and it was the one that doubted two correct answers.
A second checker isn't a more careful copy of the first. It's blind in a different place.
The browser agent reads well and quotes accurately, but it can attach a real date to the wrong rule. The critic is good at spotting missing sources and bad at knowing the law, because it's arguing from memory. The coding agent has the whole context and makes judgement mistakes: it had read the right DC page and still filed DC by category. Medical stores, so medical card. The classifier knows nothing at all, so it can't be fooled by what it thinks it knows, and catches the conclusion that doesn't follow from its own evidence.
None of them is the safety net. The net is that their errors don't overlap.
What makes it work
A few rules turned out to matter more than which models were involved.
- Every claim needs a quote and a URL. "Confirmed" without wording isn't accepted. That's what made the Tennessee mix-up visible: the quoted text didn't say what the summary said.
- "NOT FOUND" is a valid answer. If an agent has to answer, it will produce a fact. When no official source existed for a local fine or a visitor fee, it stayed off the page.
- Disagreement goes to a source, never to a vote. When the critic and the browser agent disagreed, the statute decided, not two models outvoting one.
- Nobody applies their own work. The critic can't write. The browser agent can't write. A human says "apply it", after a summary of what changes.
Each state page now lists its sources and the date it was actually checked. That date used to come from the server's clock, which made every page look verified that morning. It's now set only when someone has checked the page, and it's the date this post is about.
Top comments (4)
The load-bearing idea here is independence: the pipeline survives because the checkers fail in uncorrelated ways, so no single mistake has a clear path to the site. That's the same reason the 403/no-answer split matters — a server outside the US and a browser agent on a US connection don't just divide labor, they fail on different URLs, so coverage is a property of the ensemble rather than any one agent.
The read-only critic is the part I'd steal. Giving one Claude Code instance write access and a second instance only read access plus a voice in the thread breaks the "the agent that wrote it also grades it" trap, where a model rationalizes its own output. Separating the authority to propose from the authority to object is cheap and catches a whole class of confident-but-wrong changes.
One question on Jev as the final grader: since it can't look anything up and only judges whether the conclusion follows from the text in front of it, how do you guard against a well-sourced-looking quote that's real but stale — the Nevada "one ounce" case, where the text is internally consistent and still wrong? That seems like the gap a closed-book classifier can't close on its own.
You've found the gap. It can't, and it isn't asked to.
Jev only answers "does this conclusion follow from this text". Hand it a stale quote that agrees with itself and it will pass it every time, because nothing in the text contradicts anything. Nevada is the clean example: the old page said one ounce, and a closed-book grader handed that text would have been perfectly happy.
What caught Nevada was the fetch, not the grading. Every claim has to come with a quote from the official source as it reads now, and the page records the date that source was actually read. The stale number didn't survive because the official source, read this week, said two and a half ounces, and the quote had to match.
So staleness is handled one step earlier, by what goes into the classifier, not by the classifier. Jev's job is narrower: catching the conclusion I wrote that my own evidence doesn't support, like North Carolina. Two different failure modes, two different checks.
Independence only buys something when a step downstream actually reads the disagreement and behaves differently because of it. Your third rule is the part doing that work. Send a conflict to the statute instead of to a vote, and uncorrelated blind spots start paying. Leave that step out and you get two opinions that both clear the bar.
I read a public append-only event ledger, and it holds the negative case. Across its whole history there are 67 work items where two independent signing keys each submitted an answer, all of them translations. Byte-identical submissions: zero. Sorting the JSON keys and normalizing whitespace still leaves zero. In 61 of the 67 the two payloads do not even carry the same set of output keys. Project down to the single field holding the answer and 46 of the 67 agree, and folding case and punctuation does not move that number. The grader never looked at any of it. All 46 agreeing pairs passed on both sides, and of the 21 where the answer text differs, 17 carry a signed verdict on both sides, with 14 of those passing both. The disagreement was sitting there in machine-readable form and it never changed an outcome. That is one small log with very few signing keys, so I would not read a law about ensembles out of it.
Your critic is the piece I would re-aim rather than trust. It objected twice this week and was wrong both times, and your own third rule means an objection never decides anything anyway. It buys one more source read. So its real output is a choice about which claim gets read twice, and the hit rate of that choice this run was 0 for 2. Still a useful job. It is a different job from the one the word critic implies.
Which argues for pointing it at the claims whose source was fetched longest ago, instead of at whatever it finds implausible from memory. Memory is the one faculty Maryland and Maine showed it does not have.
That ledger is a good negative case. 21 disagreements sitting in machine-readable form, and none of them changed an outcome. That's exactly the failure the third rule exists for, and it's easy to build without noticing, because two checkers that both pass look like agreement.
You're right about the critic, too. Its two objections on the law were wrong because it was arguing from memory. What it got right was different: it flagged claims with no source behind them, like the Iowa date. So its real job isn't "is this true". It's "which claim deserves a second read", and on that it's better at spotting a missing source than a wrong one.
Pointing it at the claims whose source was read longest ago is a better rule than "whatever looks implausible to you". It gives the critic something it can actually check instead of something it has to remember. I'm going to try that on the next pass.