Five Fact-Check Gates and a Signature Passed a Page Whose Source URL Never Renders the Number

Originally published on hexisteme notes.

I run a small pipeline that drafts comparison pages and won't publish one until it clears five machine gates and I sign a fact-check sheet by hand. One page already live under that pipeline — a comparison of online card-payment fees across Stripe, PayPal, and Square — had cleared all five gates and already carried my signature. Then I ran the pipeline's dossier-export step against a real cluster and real fetching for the first time, and it rejected a citation on that already-signed, already-live page. The number in the citation was true. The page it pointed to had never rendered that number at all.

That distinction is worth holding onto, because it's tempting to read this as "the fact-check missed something" and stop there. The fact-check hadn't missed the fact — the underlying claim was correct, and the sheet recorded it correctly. What was wrong was the identity between the URL in the footnote and the text a browser actually shows when it opens that URL. Confirming that a claim is true and confirming that a specific page currently displays that claim feel like the same act once you've done the first one. They are not the same act, and nothing downstream of my signature had ever been built to do the second.

What a real fetch found that a fixture never would

The dossier-export step rejected three of the page's evidence entries outright the first time it ran against real data: their recorded quotes could not be found anywhere in the text the exporter actually fetched. One of the three was Square's entry for its international-card surcharge — an extra 1.5% on cards issued outside the United States — and the rejection reason was blunt: the quote itself was not on the page.

I opened the cited URL myself to see why. Square's fee table on that page renders client-side; a plain fetch of the page pulled back roughly ten thousand characters of navigation and FAQ text and nothing resembling a rate table. The number wasn't missing from the response, exactly — it was sitting inside the page's client-side hydration payload, the data blob a framework ships down so it can take over already-served HTML without a second round trip. The page's actual visible text ran about 3,300 characters. No shadow DOM held the number. No <template> element held it either, which would at least have been a recognized route to eventual rendering. It was present in what the server sent and absent from what a browser ever paints.

A second page on the very same site — a plain pricing page, not the one the footnote named — rendered the identical claim as ordinary static text: "An additional fee applies to payments made on credit or debit cards issued outside of the United States. 1.5%." Same fact, same number, sitting in the open. It just wasn't the page the citation pointed to.

Why five gates and a signature both said yes

None of this was a gap left open on purpose; it was a gap in what "checked" had ever meant here. The five machine gates I run check whether numbers in the body match the fact-check sheet, whether every claim in the sheet is grounded in an evidence entry, whether phrasing overlaps across pages, and whether revenue and disclosure wording follow the site's rules. Every one of those compares the drafted page against the sheet, or compares the sheet's own entries to each other. Not one of them re-opens the cited URL and asks whether the page, right now, contains what the sheet says it contains. Nothing in the five had a job that pointed outward.

I hadn't caught it by hand either, for the same underlying reason. Signing that fact-check sheet meant verifying the claim — does Square really charge more on foreign-issued cards. It does. It didn't mean following the footnote's URL and checking that the number was actually sitting there, visibly, on the page it named. Once a claim checks out as true, there's very little pull toward reopening its citation. Verifying that a source is real and verifying that the specific page you pointed to currently shows what you say it shows read, in the moment, like one task. They only turn out to be two the moment a framework's client-side rendering quietly pulls them apart, which is exactly what had happened here without anyone deciding it should.

Changing what gets compared

The fix changed the unit of comparison. Evidence used to carry a value — the number itself — checked against the drafted page's body text. Now every piece of evidence has to carry a quote, a verbatim excerpt a human can actually read, and that quote has to exist word-for-word inside whatever text got fetched from the citation's URL; the value then has to be a token drawn from inside its own quote, not a separately typed number that merely resembles it. A citation no longer passes because its number looks right. It passes because its quote is provably sitting in the page it names.

For a citation target that only renders its numbers through JavaScript, I didn't add a bypass flag. A flag that says "trust me, this one renders client-side" is a lock with no key on the other side — six months later, nobody can tell a legitimate exception from a stale one. Instead there's now an explicit option to supply the already-rendered text for a given URL directly, with the sheet recording what stood in for the live fetch: which text, its length, its hash. Whatever let a citation through stays reproducible after the fact instead of becoming someone's unverifiable say-so. For Square specifically I didn't even need that path — I repointed the footnote at the plain pricing page that already renders the number as static text. The corrected entry now reads plainly: source squareup.com/us/en/pricing, quote verification found in fetched text.

The second crack the first fix exposed

Requiring a verbatim quote-in-fetched-text match immediately broke a second citation on the same page that had been getting a free pass. PayPal's international fee, quoted at 1.50%, turned out to be two non-adjacent fragments of the source page joined with an ellipsis — someone had spliced together two separate sentences into what read like one continuous quote. Verbatim matching doesn't forgive that. The joined string, ellipsis included, exists nowhere in the fetched text.

The fix was narrower than banning ellipses outright, since a real quote sometimes does skip filler mid-sentence honestly. Each fragment on either side of an ellipsis now has to be at least fifteen characters long, has to appear verbatim in the fetched text, and the fragments have to appear in the source in the same order they appear in the quote. A splice stitching together two sentences from opposite ends of a page no longer has an ellipsis to hide behind.

Not overwriting a signed dossier

The number on the live page never changed — Square still charges exactly what the page always correctly said. What changed was which URL backs that number up, and this pipeline treats a citation swap as no smaller an edit than a change to the claim itself. I didn't patch the existing, signed dossier in place. I built a new one and ran it through export, registration, and judgment again, and the judge answered exactly the way it does for any new page: awaiting a human signature, mine, again, before it can replace what's live. There's no tier of edit small enough to skip that step.

The same day, a different corner, the same shape

A second failure surfaced that same day, in a different part of this pipeline, with nothing to do with citations — except that it was the identical failure wearing different clothes. A worker process had reported the export pipeline's exit clean, but that green result had come from a fixture cluster and a stand-in fetcher, not the real ones. Pointed at the path the tool is actually meant to run on — a real cluster, real fetching turned on — the same pipeline failed immediately, rejecting on its very first check: the real cluster had none of the revenue-axis data, source licenses, or evidence values that the fixture had always quietly supplied.

That worker's green light had been real. It just hadn't been a green light about the thing anyone actually needed it to be true of. A passed check only tells you about whatever it actually looked at, and if what it looked at was a stand-in, the pass is a statement about the stand-in. Structurally, that's the same defect as a citation whose quote had only ever been checked against the sheet that wrote it, never against the page it names — a comparison dressed up as verification that never touched the thing it claimed to verify.

What generalizes

If a check exists to verify a citation, it's worth naming, explicitly, which question it actually answers: is it comparing the source string against itself, or against what the target currently, actually renders? A check that only does the former will wave through a citation whose number is completely true and whose pointer is completely wrong, and it will do it every time, because truth and pointer accuracy are independent facts that happen to usually travel together.

Second, a green result is only as trustworthy as the path that produced it. Before treating a pass as reassurance, it's worth confirming it came from the real path — real data, a real fetch — rather than a fixture standing in for one.

Third, when a citation target only renders through client-side code, the fix isn't a flag that waives the check. It's an explicit, recorded substitute for the fetch, detailed enough to reproduce the decision later, so that whatever let a citation through stays checkable instead of becoming a permanent, silent exception.

Where this would be overkill

I wouldn't defend this at every scale. If a set of citation targets is served as plain static pages that change rarely, the URL and its rendered text are going to agree almost all the time, and forcing a fresh quote-in-fetched-text comparison on every citation could easily cost more in fetches and friction than the defects it would ever catch. This case produced a real defect specifically because the target mixed client-side hydration and static rendering across pages on the same site — a common pattern, and an inconsistent one, which is exactly the combination that lets a citation be true and its pointer be wrong at the same time without anything about the page looking obviously broken.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.