If you are building an AI agent that connects to a database, queries a knowledge base, or cites documents, you are likely dealing with a fundamental misconception: You cannot prompt-engineer an LLM into being perfectly factual.

In a recent industry survey of 800+ decision-makers, 40% cited security and compliance as the #1 barrier to scaling agentic AI. 46% specifically worry about indirect prompt injection and tool poisoning. Most teams are operating in what the report calls "leap-of-faith mode"—trusting the LLM to faithfully report what it found in a database.

I know this trap intimately. I built a local AI agent using LangGraph, Ollama, and ChromaDB connected to a live PostgreSQL ERP database. During adversarial testing, it confidently invented two fake item IDs with fake stock levels, directly contradicting the empty database result it had just received.

This is the story of how I stopped trusting the LLM to verify its own output, and built a three-layer architecture using deterministic guardrails, exact-match verification, and scope-routing to make the agent safe enough for production.

The Architecture: Prompt Establishes Intent, Code Enforces It
My first instinct was to write a massive system prompt: "Never fabricate data," "Always ground your answer in retrieved context."

It worked 90% of the time. But 90% is a failing grade in enterprise software. I had to rip the safety out of the prompt and move it into the graph state. The architecture now looks like this:

1.The Prompt: Establishes the persona and the intent to be helpful.

2.The Tool: Executes the database query or vector search.

3.The Backstop (Deterministic Code): Reads the raw tool output from the graph state and overrides the LLM's prose if a known failure shape appears.

1.The Deterministic Backstop Node

I did not use a separate validation LLM (which just adds latency and another point of failure). I used a deterministic Python wrapper that checks the state before the final response is generated.

If the database tool returns an empty array or a hard error, the code intercepts the response and replaces the LLM's output entirely:

def apply_tool_result_guards(state, response):
    """
    Deterministic backstop: if the last DB query came back empty or errored,
    do not trust the model's prose about what it found.
    """
    last_tool_message = state.get("messages", [])[-1].content

    if _is_no_data(last_tool_message) or _is_error(last_tool_message):
        return "No matching data was found for that query. The data may not " \
               "exist in the database yet, or try different filters."

    return response

The Lesson: The LLM is simply not allowed to write the final sentence about what it found if the data came back empty.

2. The Whitelist over the Correctness Checker

When asking the agent for Islamic citations, it once cited verse 5:51 (which is about interfaith alliances) for a question about usury (riba). My original regex only caught Surah X:Y formatting, so it missed (Quran 5:51).

Instead of asking the LLM to check if the citation is correct (a probabilistic, hard problem), I built a whitelist check (a deterministic, easy problem).

_ALLOWED_CITATIONS = {"2:275", "2:282"} # Pre-verified list

def enforce_citation_guard(text):
    for match in _CITATION_PATTERN.finditer(text):
        if not any(allowed in match.group(0) for allowed in _ALLOWED_CITATIONS):
            # Block even if the citation is real, if it's not on the whitelist.
            return SAFE_FALLBACK_MESSAGE 
    return text

If the model guesses a real verse that isn't on the whitelist, it gets blocked. Boring and reliable beats smart and hallucinatory.

3. Never Let the Model Do Entity-Matching

During testing, the agent correctly identified the lowest-scoring supplier from a sorted SQL query, but attached that value to the wrong row's identity. Two suppliers happened to share a display name, and the model mixed the fields.

The fix? Don't hand the LLM raw JSON and hope it reads it correctly. Modify the tool to state the answer explicitly:

# Bad: Return raw JSON
return json.dumps(rows)

# Good: Force the tool to state the entity match explicitly
return "Row 1, id=4 (Supplier: Acme Corp) has the lowest value: $4,500. Do not combine this with fields from a different row."

Layer 2: The Abstration Problem (Why 4 Local LLMs Failed)

When scaling to a 196-question benchmark, I needed the agent to refuse to answer questions that were out of scope (e.g., asking for a specific ruling that isn't in the text, or asking about modern cryptocurrencies).

I tried to use a local 7B LLM as a relevance judge. The prompt was simple: "Given this question and these 5 retrieved texts, decide if the text answers the question. If not, abstain."

I tested command-r7b-arabic, llama3.1, llama3.2:3b, and phi3. I ran 24 draws.

The result: 0 correct abstentions on out-of-corpus questions.

When asked "What is the ruling on horse meat?" (a topic that has nearby food prohibitions in the text, but no direct ruling), the models didn't say "I don't know." They grabbed the most topically-adjacent verse and confidently presented it as the answer.

This is the capability floor. LLMs are great at "Topic Matching." They are terrible at "Relevance Judgment."

The Fix: Two-Layer Scope Routing

Instead of asking the LLM to judge relevance, I split the task:

Layer 1: LLM Intent Classifier. I asked the model to classify the user's question into a category (e.g., "Is this asking for a ruling?"). LLMs are excellent at classification.

Layer 2: Regex Backstop. A deterministic keyword regex catches rule-start phrases (like "What is the ruling on..." or "How many...") to backstop the LLM.

When the gate fires, the user gets a scope message explaining that the text doesn't provide a ruling, and offers to show adjacent verses instead. It catches 8/10 out-of-scope questions with 0/4 false positives on legitimate questions.

Layer 3: The Evaluation Metric was Lying (The Reranker Bug)

For months, my lexical search bucket was stuck at ~19% recall. I blamed the reranker. I built an evaluation harness to test pre-rerank vs. post-rerank scores.

The measurement showed the opposite: The reranker was helping the Arabic path, but destroying the Roman-Urdu path by 11.7 points.

Why? Because I was asking the reranker to score (Roman-Urdu query, Arabic verse) pairs—a pairing it was never trained on. The model was scoring in a near-uniform noise band (1e-5).

The fix was to remove the reranker from the Roman-Urdu path entirely. Roman-Urdu R@5 jumped from 7.1% to 17.5% overnight.

The Final Lesson: Checkpoint Your Eval Pipeline

During this, I ran a 100-minute evaluation that crashed in the final report-generation stage because of a two-character format string error ({"n":>4d} instead of s). Because nothing was written to disk, I lost the entire 100 minutes of compute.

Always checkpoint your eval results to disk (pickle, JSON, whatever) immediately after the compute finishes, before you generate reports.

# Always checkpoint expensive compute
with open("rerank_checkpoint.pkl", "wb") as f:
    pickle.dump(rank_dicts, f)

Conclusion: How to build a reliable agent in 2026

A System Prompt is a Leash, not a Lobotomy. It establishes intent, but it cannot enforce truth.

Deterministic Backstops are Mandatory. If a tool returns an empty array or an error, override the LLM's prose entirely.

Use a Whitelist, not a Correctness Checker. Verifying if a citation is accurate is hard. Checking if it is in a pre-approved list is easy.

Don't use LLMs for Entity-Matching. If you can do the math or the sorting in Python, do it. Hand the model the answer explicitly.

Test the Test. I once had a weak assertion that passed a hallucinated quote as green because my regex only checked for the word "Hadith" nearby. A weak test is as dangerous as no test.

Building agentic AI isn't about finding the smartest model. It's about building the most resilient architecture around an inherently probabilistic system.

If you want to read the full 7-part engineering logs detailing these bugs, you can find them on my portfolio here.

By Muhammad Ahmad | AI Agent Engineer | PMP®, RMP®