AI citation checks: why a working link can't prove a quote

AI citation checks have to test the quoted words, because a link that opens only shows that a page exists at that address. The check that works is cheap: remove copy noise, require the quote to appear as one exact run of whole tokens in a stored copy of the source, and use similarity scores only to show the reader the closest passage.

What our lab found {#lab}

Local simulation with documents and bad quotes we wrote. It tests the checker, not any AI model or product.

The lab (labs/ai-10, Node.js 22, seed 42) writes ten short documents of our own and builds 60 answers, each citing one local file with one quote. Thirty quotes are copied from their file, a third of them with copy noise such as a curly apostrophe or a line break. The other 30 were built by hand: 10 real sentences cited to the wrong file, 10 merged from two half sentences, and 10 rewordings (3 of them with one changed number).

Citation links that open a real file: 60/60

All 60 cited local files existed and were readable, for the good answers and the bad ones. The lab makes no HTTP requests; in production the same blind spot belongs to an HTTP status check.

Rewordings are the hard case, because they share most of their tokens with a real sentence.

How do AI citation checks work? {#how-it-works}

Normalize both texts, then search for the quote as one contiguous run. Raw byte equality rejected 10 of 30 real quotes in the lab; answer A19 has two spaces after "When" where the page has one.

Grounded quotes verified by exact match (after normalization): 30/30
Grounded quotes that a raw, un-normalized match would wrongly reject: 10/30

The lab's normalizer applies Unicode NFKC (UAX #15), lowercases, turns punctuation into spaces and collapses whitespace. Every step can also erase meaning:

So fold typography and keep numbers as written, with sign, decimal point, exponent, superscripts and percent sign attached. The verifier also has to look just outside the matched span: in "set it to - 0.5", the quote "0.5" is a whole token of the source, and the spaced minus in front still changes the value. A number it cannot parse ("1/2", "10-20", "3:00") gets "could not check". The lab's normalizer does none of this; the verifier in Code does. Normalization also moves positions (NFKC turns "fi" into two characters), so map each token to its original span and slice the original text.

With N source tokens and M quote tokens, exact search by Knuth, Morris and Pratt is O(N + M). The closest-passage search for display is O((N - M + 1) · M): for N = 100,000 and M = 30, that is 99,971 × 30 = 2,999,130 comparisons per failed quote, so run it only inside the cited chunk. The lab's median was 0.055 ms per answer, on warmed reads of tiny local files.

Why does a 0.9 similarity score accept a changed number? {#fuzzy}

The lab's fuzzy score slides a window the length of the quote over the source and takes the best fraction of quote tokens found in it, ignoring order:

threshold  grounded-rejected  ungrounded-accepted  paraphrase-accepted  number-flips-accepted
exact      0                  0                    0                    0
0.95       0                  1                    1                    0
0.9        0                  6                    6                    3
0.8        0                  9                    9                    3
0.7        0                  11                   10                   3

At 0.9, 6 of 10 rewordings pass, all 3 number changes among them. The second column is zero on every row: looseness saved no real quote that normalization had not already saved.

Answer A59 cites cdn.txt. The page says "the origin serves only 1 request in 10"; the quote says 2. Both sentences normalize to 16 tokens, and 15 match:

score = matching tokens / quote tokens = 15 / 16 = 0.9375   (printed 0.94)
0.94 >= 0.90  ->  accepted

With k substitutions in an n-token quote, the aligned window keeps at least n - k matching tokens, so the score is at least (n - k) / n, and another window or a reordering can score higher. One substitution therefore passes at 0.9 whenever n ≥ 10. A changed number and a changed article both cost 1/n.

Citations as spans, snapshots and hashes {#design}

The strongest design copies the quote out of the source, so the model's wording never reaches the quotation marks.

1. Cite spans. Retrieval already holds each chunk's text and offsets (how RAG retrieval finds one line in 100 pages). Ask the model for chunk IDs and render the quote from stored text, using the W3C Web Annotation TextQuoteSelector and TextPositionSelector.
2. Snapshot the original. Store the fetched bytes and the extracted text, each with a SHA-256, plus extractor and normalizer versions. Only a snapshot tells a changed page from an invented quote, the copy-versus-origin problem from CDN caching.
3. Use conditional creation (S3 If-None-Match: *) to prevent accidental overwrites. For retained snapshots, configure Object Lock retention and store the object version ID. Verify the digest on every read; a hash detects change and cannot stop it.
4. Verify before display. If the model wrote the quote, check it against the snapshot, render the stored span on a pass, and drop the quotation marks on a fail.
5. Text fragments help readers locate a passage; verify against the stored snapshot before display.

An internal citation record:

{
  "citationId": "c-7f3a",
  "snapshotId": "sha256:9b1e...c04d",
  "fetchedAt": "2026-11-05T14:02:11Z",
  "url": "https://example.com/cdn",
  "extractor": "[email protected]",
  "normalizer": "tok-nfkc-lower@2",
  "hashes": { "raw": "sha256:9b1e...c04d", "text": "sha256:41aa...e9f2", "norm": "sha256:c3d0...7b18" },
  "selector": {
    "position": { "unit": "utf16", "start": 38, "end": 76 },
    "quote": { "exact": "the origin serves only 1 request in 10", "prefix": "90 percent means ", "suffix": "." }
  },
  "matches": 1
}

The offsets are JavaScript UTF-16 code units, and the unit is stored with them. A W3C TextPositionSelector counts Unicode code points, so convert UTF-16 offsets to code-point offsets when exporting one. The two counts differ after a supplementary-plane character.

What should the app show when a quote cannot be found? {#ui}

Use fuzzy matching to find the nearest passage for display, so the reader sees "1" next to "2". Label a pass "quote found": "source verified" claims more than the check proves.

Code {#code}

The verifier (labs/ai-10/quote-check.mjs) keeps each token's original span, keeps Devanagari marks and joiners, and compares numbers and operators as written, so "x = 5" does not match "x != 5". An unknown operator run such as "5!" gets "could not check", and a match is refused when a number at either end touches a sign, unit or operator just outside the span.

// quote-check.mjs  ·  node quote-check.mjs
// Tokens keep their original span. Offsets are JavaScript UTF-16 code units.
const W = String.raw`\p{L}\p{M}\p{N}‌‍`;       // ZWNJ and ZWJ stay inside words
const NUM = String.raw`[+\-−]?(?:\p{Nd}+(?:[.,]\p{Nd}+)*|[.,]\p{Nd}+)` +
            String.raw`(?:[eE][+\-−]?\p{Nd}+)?[⁰¹²³⁴⁵⁶⁷⁸⁹⁺⁻]*[%‰]?`;
// An operator run is kept whole and as written: "!=" is not "=". It is a token when it holds a
// comparison sign or sits next to a digit, so "real-time" stays prose.
const OP = String.raw`[!+\-−±/×÷*^=<>≤≥≠~≈]+`;
const SYM = String.raw`[%‰°\p{Sc}]|(?<=\p{N}\s*)${OP}|${OP}(?=\s*[.,]?\p{N})|(?=[!+\-−±/×÷*^~]*[=<>≤≥≠≈])${OP}`;
const KNOWN = new Set(['+', '-', '−', '±', '/', '×', '÷', '*', '^', '~', '≈', '=', '<', '>', '≤', '≥', '≠',
                       '==', '!=', '<=', '>=', '->', '=>', '<-']);
const TOKEN = new RegExp(
  String.raw`(?<![${W}])(?<!\p{N}[^\s${W}])(?<num>${NUM})(?![${W}])(?![^\s${W}]\p{N})` +
  String.raw`|[${W}]+|(?<sym>${SYM})`, 'gu');

function tokenize(text) {
  const out = [];
  for (const m of text.matchAll(TOKEN)) {
    const raw = m[0];
    let t;
    if (m.groups.num) t = 'n:' + raw;                        // number: compared as written
    else if (m.groups.sym)                                   // sign, unit or operator: a token;
      t = (/^[%‰°\p{Sc}]$/u.test(raw) || KNOWN.has(raw) ? 's:' : 'bad:') + raw;  // "5!", "===": bad
    else if (!/\p{N}/u.test(raw)) t = 'w:' + raw.normalize('NFKC').toLowerCase();
    else if (/\p{L}/u.test(raw)) t = 'id:' + raw;            // IPv4, x², 5ms: exact
    else t = 'bad:' + raw;                                   // 1/2, 10-20, 3:00, ½
    out.push({ t, start: m.index, end: m.index + raw.length });
  }
  return out;
}

function findAll(q, d) {                       // KMP over token arrays: O(N + M)
  const fail = [0];
  for (let i = 1, k = 0; i < q.length; i++) {
    while (k && q[i] !== q[k]) k = fail[k - 1];
    if (q[i] === q[k]) k++;
    fail[i] = k;
  }
  const hits = [];
  for (let i = 0, k = 0; i < d.length; i++) {
    while (k && d[i] !== q[k]) k = fail[k - 1];
    if (d[i] === q[k]) k++;
    if (k === q.length) { hits.push(i - q.length + 1); k = fail[k - 1]; }
  }
  return hits;
}

function closest(q, d) {                       // display only: O((N - M + 1) * M)
  let best = { score: 0, at: -1 };
  for (let s = 0; s + q.length <= d.length; s++) {
    let hit = 0;
    for (let i = 0; i < q.length; i++) if (d[s + i] === q[i]) hit++;
    if (hit / q.length > best.score) best = { score: hit / q.length, at: s };
  }
  return best;
}

// A number at either end of a match must not sit next to a sign, unit, operator
// or another number just outside the match, across spaces only: "- 0.5", "5 %", "1 / 2".
const touchy = (x) => x !== undefined && /^(n|s|bad):/.test(x.t);
const isNum = (x) => /^(n|id):/.test(x.t);
function contextOk(source, dt, h, m) {
  const first = dt[h], last = dt[h + m - 1], prev = dt[h - 1], next = dt[h + m];
  const blank = (a, b) => /^\s*$/u.test(source.slice(a, b));
  if (isNum(first) && touchy(prev) && blank(prev.end, first.start)) return false;
  if (isNum(last) && touchy(next) && blank(last.end, next.start)) return false;
  return true;
}

export function checkQuote(quote, source) {
  const qt = tokenize(quote), dt = tokenize(source);
  if (qt.length === 0) return { found: false, reason: 'empty quote' };
  if (qt.some((x) => x.t.startsWith('bad:'))) return { found: false, reason: 'could not check: numeric form' };
  const q = qt.map((x) => x.t), d = dt.map((x) => x.t);
  const hits = findAll(q, d);
  if (hits.length === 0) {
    const c = closest(q, d);
    const text = c.at < 0 ? '' : source.slice(dt[c.at].start, dt[c.at + q.length - 1].end);
    return { found: false, closest: { score: c.score, text } };
  }
  const ok = hits.filter((h) => contextOk(source, dt, h, q.length));
  if (ok.length === 0) return { found: false, reason: 'could not check: numeric context' };
  const spans = ok.map((h) => ({ start: dt[h].start, end: dt[h + q.length - 1].end }));
  const { start, end } = spans[0];
  return { found: true, start, end, original: source.slice(start, end), matches: spans.length };
}

const page = 'A cache hit ratio of 90 percent means the origin serves only 1 request in 10.';
console.log(checkQuote('the origin serves only 1 request in 10', page));
console.log(checkQuote('the origin serves only 2 request in 10', page));
console.log(checkQuote('When  the  ORIGIN serves only 1 request in 10', page));
console.log(checkQuote('x != 5', 'stop if x != 5').found);  // true: the operator matches as written

// Regression cases: every line must end in false.
const bad = [['0.5', 'set it to -0.5'], ['0.5', 'set it to - 0.5'], ['2', 'wait 20 seconds'],
             ['1.000', 'about 1,000 users'], ['5', 'set to .5'], ['5', 'a 5% loss'],
             ['5', 'a 5 % loss'], ['10 2', 'about 10² requests'], ['1 2', 'use 1/2 of it'],
             ['1', 'use 1 / 2 of it'], ['!!!', 'arbitrary source'], ['अनुपस्थित', 'अर्को पाठ'],
             ['x = 5', 'x != 5'], ['x = 5', 'x ≠ 5'], ['a = b', 'a != b'], ['x <= 5', 'x < 5'],
             ['5! ways', 'there are 5! ways'], ['x === 5', 'x === 5']];
for (const [q, s] of bad) console.log(JSON.stringify(q), JSON.stringify(s), checkQuote(q, s).found);

The first call prints found: true, start: 38, end: 76 and the original slice the UI renders. The second prints found: false with a closest passage at 0.875 (7 of 8). The third fails at 8 of 9 because "when" is not in the page; the spaces and capitals alone were forgiven. The fourth prints true, and all 18 regression lines print false. If the closest score replaced the KMP match, the second call would pass at a threshold of 0.8; at 0.9 it still fails.

Limits: lowercasing merges "US" and "us"; a Unicode minus will not match "-0.5" (a false alarm, never a false pass); and "5" quoted from "5 ms" still matches, so show surrounding text with every quote. It covers prose with numbers and simple comparisons; treat code and formulas as "could not check" until your tokenizer handles their syntax.

Limits of an exact match {#limits}

• Ellipses: "The cache does not serve stale content unless the origin is down" cut to "The cache ... serve stale content" passes a fragment check and reverses the claim. Shortened quotes need a meaning check.
• Markup and PDFs: extract with the same pipeline at retrieval and check time, strip only known artifacts such as soft hyphens, and keep ZWJ and ZWNJ, which change Devanagari spelling.
• Changed pages: if the quote matches the snapshot but is missing from a successful live extraction, investigate source or extraction changes. Show "source changed" only after confirming a content change.
• Login walls and empty JavaScript shells return 200 without the text. Treat a source too short to hold the quote as "could not check".
• A quote check does nothing about retrieved text that steers the model; see why the AI obeyed a hidden line.

Trade-offs {#trade-offs}

Our take {#our-take}

Check every quote before it appears in quotation marks. It needs no model call, only the text you retrieved, a snapshot and some string handling. We would not ship a similarity threshold as a pass condition; our semantic cache lab found the same weakness when a threshold decides reuse. A match tells you these words are in that source. Whether the source is right needs its own review, like the checks in AI evals and guardrails.

This article describes general patterns and our own local lab with synthetic documents. The bad quotes were built by hand to copy known failure shapes; the numbers show how a checker behaves and are not a hallucination rate for any AI model or product.