I froze nine yearly revisions of federal acquisition regulations into one corpus, asked 700 questions "as of" specific dates, and watched plain hybrid RAG answer from the wrong year 80% of the time. Date handling fixes it — with one honest failure mode.
Scope note, first paragraph as promised: FAR/DFARS Title 48 only, year-granularity revisions 2017–2025, US public-domain text, bge-m3 + BM25 retrieval, rule-based scoring, no LLM judge, no generator trained yet. NOT legal advice — a measurement study, not counsel.
Timeless corpus, wrong answers
Regulations change. Between 2017 and 2025, about 9,600 section-year pairs in Title 48 changed text — with a real spike around 2019–2020 (2,318 pairs). A RAG system that ignores dates retrieves across all nine years and answers with whatever ranks highest. Usually that's the newest text. For a question asked "as of" 2019, newest is wrong.
I built ClauseGym to measure exactly this: 227,449 chunks in Postgres+pgvector, each tagged with its validity range, plus a BM25 index over the same corpus and a frozen set of 700 questions generated from real diffs — 500 explicit-date ("As of July 1, 2019, what does section X say?") and 200 change questions ("What changed between 2019 and 2020?"). A sibling project measured the serving side; this one is the benchmark and the training environment, with a deterministic rule engine as the eventual reward signal.
The ladder
| retrieval | wrong year at top-1 | right version @5 |
|---|---|---|
| dateless hybrid RRF | 0.802 | 0.106 |
| year ≤ asked | 0.522 | 0.172 |
| year = asked | 0.000 | 0.332 |
| oracle section lookup | 0.000 | 1.000 |
Each stronger date constraint cuts leakage. The oracle row is a ceiling, not a system — it looks up the gold section id. The interesting row is the third: pinning retrieval to the asked year eliminates top-1 leakage and triples version accuracy.
The failure mode I kept
Strict pinning scores zero on the change questions, which legitimately need two versions. A system that pins every query to one year cannot answer "what changed." That's not a bug in the benchmark — it's the finding: date handling must be question-aware, not a blanket filter. Any RL reward built on this needs the same distinction, or the agent learns to refuse multi-version questions.
What I didn't do
No generator was trained — the GRPO loop is scaffolded, the numbers above are retrieval-only. The rule verifier covers 8 FAR clauses, not the full regulation. The M2 sanity baseline (subject + lead-word queries) favors BM25 by construction and is reported as such. All per-item results ship with the dataset so none of this is take-my-word.
Reproduce it
Corpus rebuild: ingest.py (govinfo bulk XML) → embed.py (bge-m3) → build_bm25.py → build_diffs.py → make_asof_questions.py → eval_leakage.py --arm naive,range,graph,sql. Code: https://github.com/raihan-js/clausegym — Data: https://huggingface.co/datasets/raihan-js/clausegym-bench
Top comments (0)