Correct, Cited, and Two Years Out of Date: Measuring Version Leakage in RAG

I built RAG over nine yearly revisions of federal acquisition regulations and asked it questions "as of" specific dates. Plain RAG answers from the wrong version 80% of the time — fluently, with citations. Date handling fixes it without hurting recall.

Scope note, first paragraph as promised: FAR/DFARS only, year-granularity revisions (2017–2025), US public-domain text, one 1.8B local generator, greedy decoding, no LLM judge (containment is string matching). NOT legal advice — a measurement study, not counsel.

The problem nobody benchmarks

Regulations change. The micro-purchase threshold moves, clauses get renumbered, paragraphs appear and vanish. A RAG system that treats its corpus as timeless will answer a 2022 question with 2024 text — correctly cited, completely wrong. I call it version leakage, and before this project I couldn't find an open benchmark that measures it over a corpus with official point-in-time history. So I built one: 15,485 sections × 9 revisions (90,301 versions) in Postgres with bitemporal ranges, 12,456 changed year-pairs, and 6,920 questions generated from real diffs (1,000 frozen for test, 200 flagged for hand-checking).

Date handling barely moves recall — and transforms leakage

Two columns because strict leakage overstates: a 2020 chunk served for a 2021 question counts even when the text is byte-identical. Material leakage (wrong version AND the section actually changed) is the honest number — and the ladder holds under both. The surprise is sql-first: near-perfect recall, worst material leakage. Finding the right section is easy; finding the right version is the whole game.

The generator echoes (so I measured that instead of hiding it)

First scoring pass said the agentic arm wins at answering (0.21 vs 0.03–0.16). Then I stripped each answer's own evidence text and rescored: 0.075. The 1.8B model copies chunks; the "win" was longer evidence dumps, not better answers. All arms synthesize at ≈0–0.08. I report both numbers — the retrieval ladder is the finding; generation needs a stronger model.

Two production edges, one with teeth

A text-keyed evidence cache serves wrong-version chunks 49 times out of 50 — put the date in the cache key or don't cache. And a null I kept: filtered vector search holds recall 1.0 down to 0.01% selectivity, both scan modes. The collapse others report needs bigger indexes than 101k vectors.

What it costs to serve

BM25 ~1ms, dense 0.15s, rerank 15–20s on CPU (80% of the budget — GPU it), generation 3.5s. All code, weights-free artifacts, and numbers linked below.