Imagine you're building an AI assistant for a company with thousands of internal documents. An employee asks: “Can enterprise customers cancel their contracts before renewal?”
Your system searches a vector database, retrieves a few chunks, sends them to an LLM, and gets back a confident answer. The demo works. Then someone asks the same thing differently: “What happens if a large customer wants to leave before their contract renews?”
The system retrieves a different set of documents. One is an old pricing policy, another discusses renewals, and a third only partially answers the question.
The LLM still produces a convincing response. Nothing crashed. The answer just isn't well supported. That's the point where production RAG becomes very different from a RAG demo.
If I were building a production RAG system today, I wouldn't stop at: Vector DB → LLM
That's a good starting point for proving that RAG works. For production, I'd want something closer to:
User
↓
API
↓
Query Processing
↓
Hybrid Retrieval
↓
Reranking
↓
Context Compression
↓
LLM
↓
Citation Validation
↓
Response
The important part isn't having eight separate services. It's giving each stage a clear responsibility.
Start With Query Processing
A user's question isn't always a good search query. Consider: “What are the limits for it?”
A human understands what “it” means if the previous conversation was about an enterprise API. A retrieval system looking only at those words doesn't have enough information.
Query processing can use conversation history, rewriting, metadata extraction, filters, or even deterministic rules to turn the request into something retrieval can actually work with.
For example:
The goal is simple: give retrieval a better representation of what the user actually wants.
Don't Rely on Vector Search Alone
Once we have the query, we need evidence. Vector search is great for semantic similarity, but semantic similarity isn't always relevant.
If someone searches for: “OAuth refresh token rotation” a document containing that exact phrase may be more useful than a document that is semantically similar but discusses session expiration instead. That's why I'd generally use hybrid retrieval:
┌── Semantic Search
Query ───────┤
└── Lexical Search
↓
Candidate Set
Semantic search handles conceptual similarity. Lexical search handles exact terminology, product names, identifiers, error codes, and other cases where wording matters. Together, they give us a stronger candidate set.
Retrieval Finds Candidates. Reranking Chooses.
Suppose hybrid retrieval returns 30 chunks. I wouldn't send all 30 to the LLM.
The retrieval stage should be good at finding possible candidates. A reranker can then look at those candidates against the actual query and determine which ones are most useful.
This creates an important distinction: “This document looks relevant” isn't necessarily the same as “this document helps answer the question.”
More Context Isn't Always Better
When retrieval performs poorly, one common solution is to simply retrieve more. Five chunks become twenty. Twenty become fifty.
Eventually, the model gets a huge prompt containing everything vaguely related to the question. That can make the system worse.
Irrelevant context consumes tokens, increases latency, and can bury the evidence that actually matters. That's where context compression becomes useful.
Instead of giving the LLM every retrieved chunk, we can reduce the context to the passages that are actually useful for answering the question. The objective isn't: Give the model everything we found. It's: Give the model the evidence it needs.
Let the LLM Reason Over Evidence
Only after retrieval, reranking, and context construction would I let the LLM generate the answer. Its job should be reasoning over the evidence we've selected, not fixing a broken retrieval system.
If retrieval returns the wrong information, switching to a larger model isn't a reliable solution.
A stronger model can actually make the problem harder to notice because it can produce a more convincing answer from bad evidence.
Generation Shouldn't Be the Final Trust Boundary
This is where I'd add another layer: citation validation. Suppose the model says: “Enterprise customers can cancel 30 days before renewal.” Where did the 30 days come from?
If none of the retrieved evidence supports that claim, the answer is still wrong regardless of how confidently it was written. A validation layer can check whether important claims are actually supported by the available evidence.
The implementation could range from simple citation checks to dedicated verification models, depending on how much risk the application carries. The important architectural principle is: A generated citation isn't automatically a valid citation.
The Real Value Is Debuggability
The biggest advantage of this architecture isn't just better answers. It's that you can actually debug failures. If the system gives a bad answer, you can ask:
- Did query processing misunderstand the request?
- Did retrieval miss the correct document?
- Did reranking choose the wrong evidence?
- Did context compression remove something important?
- Did the LLM misinterpret the evidence?
- Did citation validation fail to catch an unsupported claim?
With a simple Vector DB → LLM architecture, all of those problems tend to collapse into one vague question: “Did the model hallucinate?”
That's not enough to operate a production system. The interesting part of production RAG isn't the vector database or the LLM individually. It's the entire path: Query → Evidence → Answer → Verification
Not every application needs every layer, but I'd want those responsibilities to be explicit. A vector database can retrieve information. An LLM can reason over information.
Neither one, by itself, gives you a reliable RAG system. The model is one component. The retrieval system is the product.
Top comments (0)