This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content
What I Built
My agent kept citing real documents for words those documents never contained. Not hallucinated quotes: real sentences, attributed to the wrong source. So it now proves every quoted phrase against the exact document it cites, straight from the dataset, and marks the ones that fail.
TSB Oracle answers car repair questions from the public record, and shows you where that record contradicts itself.
Ask why your car brakes on its own and you will find a service bulletin naming a cause, a revision of that bulletin a week later that quietly drops the sentence, a federal investigation covering more model years than the fix does, and owners saying the dealer called it normal. All of it is true at once. A search engine hands you whichever page ranks best, and a chatbot picks one and sounds certain.
This agent retrieves all of it. Every sentence carries the NHTSA id it came from, quotes are checked against the document they are attributed to, and when two sources disagree about your exact car, the disagreement is put in front of you instead of resolved by guesswork.
The dataset is one vehicle and one system, imported from NHTSA's public API: unexpected automatic emergency braking on the 2017 to 2022 Honda CR-V. The conflict at the centre of it is real. NHTSA's investigation EA24-002 covers 2017 to 2022. Honda's software update 26-091 stops at 2019 and excludes the LX trim. A 2021 owner is inside an open federal investigation with no published remedy, and nothing in the record says one is coming.
What the agent does with that:
- cites every factual claim with the NHTSA id, linked to the public record
- flags in amber any citation it did not actually retrieve
- verifies quoted words against the dataset's own copy of the cited document
- shows contradictions side by side, with dates and a plain explanation of why they conflict
- proposes how a disagreement resolves, and waits for a person to approve it
Demo
The demo car is a 2021 Honda CR-V EX. The vehicle in the header is editable, so these all work:
| Vehicle | Ask | What it shows |
|---|---|---|
| 2021 CR-V EX | My CR-V brakes hard on its own with nothing ahead. The dealer says that's normal. Is there a fix? | the coverage gap, already settled by review |
| 2018 CR-V LX | Does Service Bulletin 26-091 apply to my car? | a trim exclusion |
| 2018 CR-V EX | Honda and NHTSA disagree about what causes the braking. Which applies to my car? | an open contradiction, and a proposal |
| 1994 Civic del Sol | Are there any recalls for this car? | no sources, and it says so |
While it works, it shows what it is reading rather than a spinner:
Every question is a live model call against my own API balance, so if the demo stops answering, that is why.
Code
github.com/pyarchana/tsb-oracle, tagged v1.0.
How I Used Sanity
The content model. Four types do the work. tsb is a source document, a bulletin, dealer message, investigation or owner complaint, with its model years, excluded trims and a link to the NHTSA record. claim is one statement with the exact words it rests on. contradiction joins two claims that disagree and explains why. decision records how a contradiction was resolved, who resolved it, and whether a person has approved it yet.
What I pointed Context at. The Knowledge Base is built from the tsb and claim types, with the claim's source reference unfolded so each claim carries its document's id. I deliberately left contradiction out. If I had included it, the Knowledge Base would have read my conflicts back to me. Excluding it meant Context had to find disagreement in the sources on its own.
The tools. The agent gets initial_context and knowledge_base_read over the Context MCP endpoint, plus two of my own:
-
check_applicabilityruns GROQ over the dataset and returns which documents cover this year, make, model and trim, which contradictions touch them, and whether any has been settled. It answers questions the Knowledge Base cannot: is this trim excluded, is this VIN in range, is this argument already over. -
record_decisionwrites a proposal, never a settled decision.
So Context supplies the wording and GROQ supplies the structure. The Knowledge Base tells the agent what a bulletin says; the dataset tells it whether that bulletin applies to the car in front of it.
What the Knowledge Base build caught. Four issues, all real. NHTSA's own summary of bulletin A18-006 writes "OTC" where the codes are DTCs and "MIO" where the display is the MID. Two entries merged NHTSA's complaint counts with its per-vehicle totals: 31 crashes alleged in complaints against 47 across all reports, 50 injuries against 93. I resolved each in favour of what the source actually supports, and resolutions carry into every later build.
What it did not catch, and why I left that alone. None of my curated claims conflicted. Every one of them says who made the statement it records, so "version 1 of the bulletin says X" does not flatly contradict "version 2 says Y". I could have written an unattributed claim to make the issues feature fire. Staging a conflict to demo conflict detection would have proved nothing.
Why the knowledge base cannot back a quote I assumed Knowledge Base entries could back up a quotation. They cannot, and should not. An entry is a synthesis: it merges several documents into one topic, paraphrases them, and attaches footnotes. That is right for retrieval and wrong for quoting, because a phrase can appear in an entry that no source ever used. So the app checks quoted words against the dataset's verbatim copy of the cited document instead. On a live run it caught the agent quoting version 1 of a bulletin and citing version 2, which is exactly the kind of mistake that sends someone to a dealer asking for the wrong repair.
Persistence. When the agent proposes a resolution, it writes a decision with status proposed. A person opens it in the Studio and approves or rejects it. Approving flips the decision and its contradiction in one transaction, so the two can never disagree about whether a question is settled. From then on, the next person asking about that vehicle gets the settled answer and its rationale instead of the argument.
That review step exists because the demo is public. Without it, one conversation could settle a question for everyone who asks after it.
Sanity Project Details
Project ID: e72p6sym, dataset production, which is public.
A live query against it, no token needed:
https://e72p6sym.api.sanity.io/v2026-09-01/data/query/production?query=*[_type=="tsb"]{tsbNumber,title,sourceType,status,modelYears}
The Studio is embedded at /studio in the app, and the schemas are in sanity/schemaTypes.
Top comments (7)
Archana, the decision-in-one-transaction detail is the part worth stealing. Flipping the decision and its contradiction together so they can't disagree is exactly the kind of invariant that's easy to skip under deadline pressure and expensive to debug once two records silently drift apart. The two-tier split (Knowledge Base for wording, GROQ for structure) is cleaner than most retrieval setups bother with too. Did deliberately excluding contradiction from the Knowledge Base ever tempt you to peek at it while debugging, or did that discipline hold the whole build?
It held though less through willpower than because it cost me nothing. The agent still sees every disagreement, it just gets them from a direct query over the dataset rather than from the Knowledge Base. Keeping them out only meant Context had to notice conflicts on its own instead of reading mine back to me.
The real temptation came from somewhere else. The first build found none of my conflicts and the quick fix would have been to feed them in so the feature had something to show. I left it alone. Every claim I wrote says who said it, and two attributed statements do not flatly contradict each other. What the build did find were four genuine errors in NHTSA's own summaries, which was worth more to me than a conflict I had staged for myself.
On the two records moving together what happened was that the failure I was picturing is an approved decision sitting beside a question still marked open, and the agent reopening something a person had already settled. Writing both in one go was the cheapest way to make that impossible.
Archana, that distinction clicked for me - an agent noticing a conflict on its own from a direct query is a genuinely different (and better) claim than "the tool fed it the conflict and it repeated that back." I hadn't separated those two in my head until you put it that way.
And the point about the four genuine NHTSA summary errors being worth more than the conflict you'd staged is the real finding here, more than the two-records-in-one-transaction fix - a test that surfaces mistakes in the data you trusted going in is doing something a synthetic red-team case can't.
This matches what we see operating a model-routing layer (32 models, one key at heypico.ai): the deterministic scaffolding around the LLM is what makes multi-model setups viable. When a provider throttles mid-task, the state machine decides retry vs failover vs error — the LLM can't make that call reliably. Debugging a 'flaky agent' is usually debugging a missing state machine around a fine model.
Agreed, and that's why the checks sit in code rather than in the prompt. The model is good at reading the record and saying what it found; the guarantees come from the layer around it. Quotes are verified against the source document before they reach the screen, and the tool that records a resolution only accepts a claim that actually belongs to the disagreement in question.
That layer pays for itself. In testing it caught a quote paired with the wrong version of a bulletin, both documents real and both retrieved, and marked it in the answer instead of letting it pass. The model still makes the judgment about which side applies to a given car, and a person signs it off before it becomes the settled answer for whoever asks next.
The quote check against the verbatim document is the strongest part of this. One regression test I would add: pin each quote to a document revision or content hash as well as its NHTSA ID, then keep a fixture where bulletin v1 and v2 differ by just one sentence. That catches a quote that is real but belongs to a different revision even if both versions were retrieved. Do you preserve the imported source revision when the dataset is refreshed?
This is the right split: retrieval hit ≠ faithfulness.
I’ve seen the same failure mode you describe — a real document in the cite list, and a sentence that document never contained. Ranking/relevance can still look fine while the quote-span check fails.
The gate I’d keep separate from “did we retrieve something useful?” is: for every quoted span, verify membership against the cited source id (fail closed on miss). That is a retrieve-first evidence check, not another generation polish pass.