How RAG Actually Works: Chunking, Embeddings, Retrieval, and Reranking Under the Hood

RAG feels simple on paper. Split documents into chunks, embed them, fetch the closest ones at query time. Done.

That description hides half a dozen hard sub-problems. Split too coarsely and you retrieve paragraphs that bury the relevant sentence in noise. Split too finely and no single chunk carries enough context to be useful. Use only dense retrieval and you miss documents where an exact product code or rare proper noun is the critical signal. Use only keyword search and you miss paraphrases. Retrieve top-k blindly and the final context window fills with redundant or marginally relevant chunks.

Most tutorials stop at "embed, store, retrieve." This article goes deeper — what actually happens at each stage, where the trade-offs live, and what production systems do differently.

The Pipeline at a Glance

INGESTION (offline, whenever documents change)
sources → parse/clean → deduplicate → chunk → embed (dense) + index terms (BM25) → store vectors, text, metadata

QUERY (online, per request)
question → (optional) rewrite/expand → hybrid retrieval (BM25 + dense, top 50–200) → rerank with cross-encoder (keep top 5–10) → select for relevance/diversity within token budget → build prompt with numbered sources → generate with citation instructions → check citations → return answer with links

Two pipelines meet at the index. The offline path runs when documents change. The online path runs per request. Keeping them aligned is where most RAG systems break.

1. Chunking: The Hardest Part Nobody Talks About

Every embedding model has a maximum token length — 512 tokens for older bi-encoders, 4,096–8,192 for modern ones (text-embedding-3-large, nomic-embed-text-v1.5). But even if your model supports 8k tokens, shoving an entire chapter into a single vector is a mistake: the embedding averages the semantics of every sentence, so a query about one narrow fact scores poorly against a dense multi-topic chunk that happens to contain that fact.

Chunking controls a fundamental quality–recall trade-off:

• Too large chunks — high recall (the fact is somewhere in there), but low precision (the LLM gets noise and may hallucinate or ignore the signal)
• Too small chunks — high precision, but context loss: "It increased by 12%" is meaningless without its referent

Getting chunking wrong is probably the single most common cause of poor RAG performance in production.

Fixed-Length Chunking with Overlap

The simplest approach: split every chunk_size tokens, with an overlap of tokens shared between adjacent chunks so sentence boundaries don't cut off context.

def fixed_chunk(text: str, chunk_size: int = 512, overlap: int = 64, tokenizer=None) -> Iterator[str]:
    if tokenizer is None:
        tokens = text.split()
        decode = lambda ids: " ".join(ids)
    else:
        tokens = tokenizer.encode(text)
        decode = tokenizer.decode

    step = chunk_size - overlap
    if step <= 0:
        raise ValueError("chunk_size must be strictly greater than overlap")

    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        yield decode(tokens[start:end])
        if end == len(tokens):
            break
        start += step

Typical production values: chunk_size=256–512 tokens with overlap=32–64. The overlap parameter is crucial — without it, a sentence split across two chunks loses its grammatical antecedent in both halves.

Semantic Chunking

Fixed-length chunking is agnostic to content structure. Semantic chunking exploits embedding similarity to find natural topic breaks:

1. Split text into sentences
2. Embed each sentence
3. Compute cosine similarity between consecutive sentence embeddings
4. Mark a chunk boundary wherever similarity drops below a threshold (or is a local minimum)

def semantic_chunk(sentences: list[str], embed_fn, threshold: float = 0.75, min_sentences: int = 3) -> list[str]:
    embeddings = embed_fn(sentences)  # shape (N, D)
    sims = [cosine_similarity(embeddings[i], embeddings[i+1]) for i in range(len(sentences)-1)]

    chunks = []
    current = []
    for i, sent in enumerate(sentences):
        current.append(sent)
        if i < len(sims):
            is_boundary = sims[i] < threshold and len(current) >= min_sentences
            if is_boundary:
                chunks.append(" ".join(current))
                current = []
    if current:
        chunks.append(" ".join(current))
    return chunks

Semantic chunking produces more coherent chunks but is slower (requires an embedding call) and more sensitive to the threshold. A common tuning strategy: run the chunker on a held-out set, visualize the similarity curve, and set the threshold at the 15th percentile of observed similarities.

Structure-Aware Chunking

Real documents have structure: headings, paragraphs, bullet lists, code blocks. Respecting that structure almost always outperforms purely statistical methods.

# The production equivalent: LangChain's MarkdownHeaderTextSplitter
from langchain_text_splitters import MarkdownHeaderTextSplitter

md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)
chunks = md_splitter.split_text(markdown_document)
# Each chunk carries heading metadata for downstream filtering

The libraries you actually ship. Nobody writes chunkers from scratch twice — you write them once to understand the trade-off, then use maintained splitters. The two dominant ones are LangChain (langchain_text_splitters) and LlamaIndex (llama_index.core.node_parser), and both express exactly the strategies above.

Late Chunking (The Context-Preserving Trick)

Late chunking, introduced by Günther et al. (2024) and popularized with jina-embeddings-v2, inverts the usual order:

1. Pass the entire document through the transformer encoder to produce contextualized token embeddings
2. Mean-pool over the token spans corresponding to each logical chunk

The attention mechanism lets every token see its full document context before the pooling boundary is applied. "It increased by 12%" gets an embedding that encodes what "it" refers to, because self-attention already saw the preceding paragraph.

# late_chunking.py — illustrative
from transformers import AutoTokenizer, AutoModel
import torch

def late_chunk_document(document: str, chunk_boundaries: list[tuple[int, int]], model_name: str = "jinaai/jina-embeddings-v2-base-en") -> list[LateChunk]:
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
    model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()

    encoding = tokenizer(document, return_tensors="pt", return_offsets_mapping=True, truncation=True, max_length=8192)
    offset_mapping = encoding.pop("offset_mapping")[0]
    input_ids = encoding["input_ids"]
    attention_mask = encoding["attention_mask"]

    with torch.no_grad():
        token_embeddings = model(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state[0]

    chunks = []
    for start_char, end_char in chunk_boundaries:
        token_mask = (offset_mapping[:, 0] >= start_char) & (offset_mapping[:, 1] <= end_char) & (offset_mapping[:, 1] > offset_mapping[:, 0])
        span_embeddings = token_embeddings[token_mask]
        if span_embeddings.shape[0] == 0:
            continue
        chunk_embedding = span_embeddings.mean(dim=0)
        chunk_embedding = chunk_embedding / (chunk_embedding.norm() + 1e-9)
        chunks.append(LateChunk(text=document[start_char:end_char], embedding=chunk_embedding, start_char=start_char, end_char=end_char))
    return chunks

Constraint: the full document must fit within the model's context window (~8k tokens). For longer documents, apply late chunking within sliding windows.

Late chunking has a generative cousin: contextual retrieval (Anthropic, 2024) asks an LLM to write a one-sentence situating preamble for each chunk and prepends it before embedding. Late chunking is cheap (one encoder pass, no LLM) but needs a long-context encoder; contextual retrieval works with any encoder but costs one LLM call per chunk at index time. They compose.

2. Hybrid Search: BM25 + Dense Retrieval

Dense retrieval excels at semantic similarity but struggles with exact-match recall. If a user asks about "CVE-2023-44487" or a specific product SKU, the query embedding may not land near the document containing that exact string — because the model generalizes. BM25 handles these cases natively.

BM25 score for query q against document d:

BM25(q,d) = Σ IDF(t) * (f(t,d) * (k1+1)) / (f(t,d) + k1 * (1 - b + b * |d|/avgdl))

where f(t,d) is term frequency, |d| is document length, avgdl is average document length, k1≈1.2–2.0, b≈0.75.

Use a real BM25 engine. A naive Python implementation rescores the entire corpus on every query — O(N⋅|q|) — fine for teaching, hopeless beyond a few thousand chunks. Real lexical retrieval uses an inverted index: for each query term, walk only the posting list of documents that actually contain it.

Open-source ladder:

• bm25s — sparse-matrix BM25 in Python; right default for single-process index up to millions of short chunks
• Pyserini — Python bindings over Lucene; research-grade path
• Elasticsearch / OpenSearch — distributed inverted index when corpus outgrows one machine; both ship native hybrid retrieval and built-in RRF
• Most vector databases now do the sparse leg too: Qdrant, Weaviate, Milvus all accept sparse vectors alongside dense and expose server-side RRF fusion

Reciprocal Rank Fusion (RRF)

Given a BM25 ranking and a dense ranking, how to combine them? Cormack et al. (2009) introduced RRF:

RRF(d) = Σ 1 / (k + rank_r(d))

where k = 60 (smoothing constant). Documents not in a ranking are omitted from that term. Highest combined RRF score wins.

RRF does not require score normalization — it only uses rank ordinals, making it immune to the scale mismatch between BM25 scores (0–20) and cosine similarities (0.5–1.0).

def reciprocal_rank_fusion(rankings: list[list[tuple[int, float]]], k: int = 60) -> list[tuple[int, float]]:
    rrf_scores = defaultdict(float)
    for ranked_list in rankings:
        for rank, (doc_idx, _score) in enumerate(ranked_list, start=1):
            rrf_scores[doc_idx] += 1.0 / (k + rank)
    return sorted(rrf_scores.items(), key=lambda x: -x[1])

Worked example: BM25 ranks [D2, D1, D3, D4, D5], dense ranks [D1, D3, D2, D5, D4]. With k=60:

• D1: 1/62 + 1/61 = 0.0325 → rank 1 (wins despite BM25 rank 2)
• D2: 1/61 + 1/63 = 0.0323 → rank 2
• D3: 1/63 + 1/62 = 0.0320 → rank 3

D1 wins in fused ranking because its dense rank of 1 contributes a large term. The k=60 constant prevents any single top-1 ranking from dominating completely.

Learned Sparse Retrieval (SPLADE)

BM25 and dense retrieval fail in opposite directions. Learned sparse retrieval keeps the representation sparse (a vector over the tokenizer's vocabulary, lives in an inverted index) but learns the weights with a transformer instead of counting term frequencies.

SPLADE (Formal et al., SIGIR 2021) runs text through a masked-language-model head, producing a logit for every vocabulary term at every position. Then saturate and max-pool over positions:

w_j = max_i log(1 + ReLU(w_ij))

The log(1+·) is a saturation function borrowed from BM25's term-frequency damping. Sparsity is trained in with a FLOPS regularizer — typical passage keeps ~100 non-zero terms.

Payoff: term expansion. A passage about "myocardial infarction" gets non-zero weight on heart and attack even though those strings never appear, so a lexical index can now match a paraphrase.

# sentence-transformers v5+ added SparseEncoder
from sentence_transformers import SparseEncoder

model = SparseEncoder("naver/splade-v3")
doc_emb = model.encode_document(corpus)       # sparse tensors, |V|-dimensional
query_emb = model.encode_query("http2 rapid reset attack")
scores = model.similarity(query_emb, doc_emb) # sparse dot product

# Inspect *why* a document scored
print(model.decode(doc_emb[0], top_k=8))  # e.g. [('http', 2.1), ('reset', 1.9), ...]

SPLADE is a first-stage retriever — drop-in replacement for (or third leg alongside) BM25. Fuse its ranking with dense ranking using RRF, then rerank with cross-encoder. Costs are real: encoding requires a transformer forward pass at both index and query time, and expansion terms lengthen posting lists. The honest 2026 summary: BM25 + strong dense encoder remains the pragmatic default; learned sparse is the upgrade when exact-match recall matters and queries are paraphrase-heavy, or when you need interpretability without giving up semantic matching.

3. Cross-Encoder Rerankers: The Precision Stage

The retrieval stage (BM25 or dense) must be fast — processing millions of documents in tens of milliseconds. Speed requires a bi-encoder architecture: query and document embedded independently, similarity = cheap dot product. Downside: query and document tokens never "see" each other during encoding, so nuanced relevance signals (multi-hop, contrastive relevance) can be missed.

A cross-encoder receives the concatenation [query; document] as a single sequence and produces a scalar relevance score. Every query token attends to every document token, enabling fine-grained relevance reasoning at the cost of O(N) forward passes (one per candidate). This makes cross-encoders unsuitable for first-stage retrieval but ideal for reranking a small shortlist of 20–100 candidates.

There's a well-defined middle: late interaction (ColBERT) keeps one vector per token rather than one per document, precomputes all document token vectors offline like a bi-encoder, and scores with a MaxSim operator recovering much of the cross-encoder's token-level matching. Costs ~10–100x more index storage in exchange for cross-encoder-like quality at first-stage speed.

Two-Stage Pipeline

Query → Hybrid Retrieval (BM25 + dense) → top-100 candidates
          fast, cheap recall
      → Cross-Encoder Reranker → scores all 100 [query; passage] pairs
          slower, expensive, precise scoring
      → top-5 chunks → LLM Context Window

Only the 5 winning chunks reach the LLM context window, keeping prompt costs low while the cross-encoder's joint query-passage attention ensures relevance quality.

from sentence_transformers import CrossEncoder
import numpy as np

def load_reranker(model_name: str = "cross-encoder/ms-marco-MiniLM-L-6-v2") -> CrossEncoder:
    return CrossEncoder(model_name, max_length=512)

def rerank(query: str, candidates: list[str], model: CrossEncoder, top_k: int = 5, batch_size: int = 32) -> list[tuple[str, float]]:
    pairs = [(query, passage) for passage in candidates]
    scores = model.predict(pairs, batch_size=batch_size)
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return ranked[:top_k]

# Latency numbers (A100):
# cross-encoder/ms-marco-MiniLM-L-6-v2, 100 candidates, avg 200 tokens: ~60ms
# On CPU (Intel Xeon): ~400ms for same 100 candidates

Practitioner tip: Two APIs, one job. Classical BERT-style rerankers (ms-marco-*, bge-reranker-*, jina-reranker-*) use CrossEncoder. The 2025–2026 generation of LLM-initialized rerankers — Qwen3-Reranker, mxbai-rerank-v2 — read relevance from a causal LM as the logit of a "yes"/"no" token given a prompt with instruction, query, and document. They buy prompt conditioning ("rank by whether the passage contains a numeric answer") and long context, and cost a full decoder forward pass per pair. Load with AutoModelForCausalLM per model card, or use a wrapper: rerankers (AnswerDotAI) gives one Reranker(...).rank(query, docs) call over cross-encoders, ColBERT, LLM rerankers, and hosted APIs alike. In production, serve the reranker off the request path — HuggingFace Text Embeddings Inference hosts reranker models behind a /rerank endpoint with continuous batching.

Training Your Own Reranker

If you have domain-specific relevance labels (click logs, human annotations, distillation from a more powerful model), fine-tune a cross-encoder with binary cross-entropy on (query, positive, negative) triples, or listwise ranking loss (LambdaLoss, ListNet).

from datasets import Dataset
from sentence_transformers.cross_encoder import CrossEncoder, CrossEncoderTrainer, CrossEncoderTrainingArguments
from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss

train_dataset = Dataset.from_dict({
    "query": ["What is RRF?", "What is RRF?"],
    "passage": [
        "Reciprocal rank fusion combines ranked lists using 1/(k+rank).",
        "The capital of France is Paris.",
    ],
    "label": [1.0, 0.0],  # 1 = relevant, 0 = hard negative
})

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", num_labels=1)
loss = BinaryCrossEntropyLoss(model)

args = CrossEncoderTrainingArguments(
    output_dir="./my-domain-reranker",
    num_train_epochs=1,  # 1 epoch usually enough for domain adaptation
    per_device_train_batch_size=16,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    fp16=True,
)

trainer = CrossEncoderTrainer(model=model, args=args, train_dataset=train_dataset, loss=loss)
trainer.train()
model.save_pretrained("./my-domain-reranker/final")

The data matters more than the loss. Pointwise BCE on random negatives teaches almost nothing — random negatives are trivially separable. Mine hard negatives: run your existing first-stage retriever, take documents ranked 10–100 that are not labelled relevant, and use those as zeros — they are exactly the confusions the reranker exists to fix. Usual ratio: 4–8 hard negatives per positive. With graded relevance or teacher scores, prefer listwise loss (LambdaLoss, ListNet) over BCE: it optimizes the ordering of a candidate list, which is what nDCG actually measures. Distilling a large reranker's scores into a MiniLM-sized student with MarginMSELoss is the standard way to get most quality at a fraction of latency.

4. Query Rewriting and HyDE

The query the user types is rarely optimal for retrieval — short, colloquial, or implicit.

Multi-Query Expansion

Generate several paraphrases of the original query, retrieve for each, and union the result sets (deduplicating by document ID). Dramatically improves recall for queries that can be stated multiple ways.

REWRITE_PROMPT = """You are a retrieval expert. Given a user question, produce {n} alternative phrasings that together cover the semantic space of the question. Return a JSON list of strings.

User question: {question}

Alternative phrasings (JSON list):"""

def expand_query(question: str, llm_call: callable, n_variants: int = 3) -> list[str]:
    prompt = REWRITE_PROMPT.format(question=question, n=n_variants)
    response = llm_call(prompt)
    # extract JSON list from LLM output...
    return [question] + variants[:n_variants]  # include original

HyDE — Hypothetical Document Embeddings

Instead of embedding the (short, vague) query, ask the LLM to generate a hypothetical document that would answer the query, then embed that document and retrieve using its embedding.

The intuition: a hypothetical answer document and a real answer document inhabit closer regions of embedding space than the query and the answer document, because they share vocabulary, entity mentions, and syntactic structure.

HYDE_PROMPT = """Write a short, factual paragraph (2-4 sentences) that directly answers the following question. Do not say "I don't know" — write the best answer you can, even if uncertain.

Question: {question}

Answer:"""

def hyde_embed(question: str, llm_call: callable, embed_fn: callable, n_hypotheses: int = 1) -> np.ndarray:
    hypotheses = []
    prompt = HYDE_PROMPT.format(question=question)
    for _ in range(n_hypotheses):
        hypotheses.append(llm_call(prompt).strip())
    embeddings = embed_fn(hypotheses)
    mean_embedding = embeddings.mean(axis=0)
    return mean_embedding / (np.linalg.norm(mean_embedding) + 1e-9)

HyDE hallucination risk: The LLM's hypothetical document will contain plausible-sounding but possibly incorrect facts. This is fine for retrieval (you only use the embedding, not the text), but be careful not to accidentally include the hypothesis in the LLM context. Always retrieve from the real corpus using the HyDE embedding, then pass the retrieved real documents to the generator.

Use HyDE when: corpus uses formal/technical language differing from query style; queries are short and ambiguous; you can afford 1–2 extra LLM calls per query (~200ms latency budget). Avoid when: queries are already long and specific; LLM hallucinations could mislead retrieval; low-latency SLA (<50ms) prevents extra LLM calls.

5. Metadata Filtering and Parent-Child Retrieval

Purely semantic retrieval treats all documents as equally eligible. In practice, queries have hard constraints: "only Q4 2024", "only 'legal' category", "only product model XYZ-500". Metadata filtering applies these constraints before or during ANN search, drastically reducing the candidate pool and improving precision.

Pre-filtering vs Post-filtering

• Pre-filter (filter before ANN): restrict index to documents matching metadata predicate, then search that subset. Exact, but gives up prebuilt ANN graph over surviving set.
• Post-filter (retrieve top-k, then filter): fast, but may need to retrieve large k to guarantee filtered results cover top-k meaningful hits.

Modern vector databases (Qdrant, Pinecone, Weaviate, Milvus) support hybrid pre/post-filtering with HNSW graph filterable attributes.

# Qdrant example
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
import numpy as np

def filtered_search(client, collection_name, query_vector, filters: dict, top_k=10):
    conditions = []
    if "category" in filters:
        conditions.append(FieldCondition(key="category", match=MatchValue(value=filters["category"])))
    if "date_after" in filters or "date_before" in filters:
        conditions.append(FieldCondition(key="timestamp", range=Range(gte=filters.get("date_after"), lte=filters.get("date_before"))))

    qdrant_filter = Filter(must=conditions) if conditions else None
    response = client.query_points(
        collection_name=collection_name,
        query=query_vector.tolist(),
        query_filter=qdrant_filter,
        limit=top_k,
        with_payload=True,
    )
    return [hit.payload for hit in response.points]

Parent-Child Document Retrieval

Index fine-grained child chunks (single sentences or 128-token windows) for high-precision retrieval, but when a child chunk is retrieved, return its parent document (full paragraph or section) as LLM context. Gives the LLM surrounding context while keeping the retrieval signal sharp.

@dataclass
class ChildChunk:
    id: str
    parent_id: str
    text: str
    embedding: np.ndarray

@dataclass
class ParentDocument:
    id: str
    text: str
    metadata: dict

def parent_child_retrieve(query_embedding, child_index, parent_store: dict, top_k=5) -> list[ParentDocument]:
    child_hits = child_index.search(query_embedding, top_k=top_k * 3)  # over-fetch for unique parents
    seen_parent_ids = set()
    parents = []
    for child in child_hits:
        pid = child.parent_id
        if pid not in seen_parent_ids:
            seen_parent_ids.add(pid)
            if pid in parent_store:
                parents.append(parent_store[pid])
        if len(parents) >= top_k:
            break
    return parents

Both frameworks ship this: LangChain's ParentDocumentRetriever, LlamaIndex's SentenceWindowNodeParser (retrieve a sentence, return window of ±k sentences) and AutoMergingRetriever (retrieve leaf chunks over hierarchy, substitute parent when enough siblings hit — the behavior worth stealing if you build your own).

One thing this doesn't fix: redundancy. Five distinct chunks from five near-identical documents will still all survive into the prompt. Standard remedy: Maximal Marginal Relevance (MMR) — greedily trade query relevance against novelty relative to already-selected chunks — applied after reranking and before prompt assembly.

6. A Production RAG Pipeline

@dataclass
class RAGConfig:
    chunk_size: int = 512
    chunk_overlap: int = 64
    chunking_strategy: str = "fixed"  # "fixed" | "semantic" | "structural"
    bm25_top_k: int = 40
    dense_top_k: int = 40
    rrf_k: int = 60
    use_hyde: bool = False
    n_query_variants: int = 1
    use_reranker: bool = True
    reranker_model: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"
    rerank_candidates: int = 50
    reranker_top_k: int = 5
    metadata_filter: dict = field(default_factory=dict)

def run_rag_pipeline(query: str, bm25_index, dense_index, embed_fn, llm_fn, reranker, config: RAGConfig) -> RAGResult:
    # Step 1: Query representation
    if config.use_hyde:
        query_embedding = hyde_embed(query, llm_fn, embed_fn)
    else:
        query_embedding = embed_fn([query])[0]

    # Step 2: Hybrid retrieval
    bm25_results = bm25_index.search(query, top_k=config.bm25_top_k)
    dense_results = dense_index.search(query_embedding, top_k=config.dense_top_k)
    fused = reciprocal_rank_fusion([bm25_results, dense_results], k=config.rrf_k)
    fused_chunks = [(all_docs[idx].text, score) for idx, score in fused]
    candidate_texts = [t for t, _ in fused_chunks]
    rrf_by_text = dict(fused_chunks)

    # Step 3: Reranking
    if config.use_reranker and reranker is not None:
        reranked = rerank(query, candidate_texts[:config.rerank_candidates], model=reranker, top_k=config.reranker_top_k)
        final_chunks = [t for t, _ in reranked]
        reranker_scores = [float(s) for _, s in reranked]
    else:
        final_chunks = candidate_texts[:config.reranker_top_k]
        reranker_scores = []

    return RAGResult(query=query, retrieved_chunks=final_chunks, rrf_scores=[rrf_by_text[t] for t in final_chunks], reranker_scores=reranker_scores)

Choosing Configuration for Your Domain

7. Diagnostics and Iteration

A RAG pipeline has many knobs. Easy to tune one component in isolation while unknowingly degrading another. Rigorous approach: evaluate each stage separately with stage-specific metrics, then measure end-to-end quality.

from typing import NamedTuple

class RetrievalMetrics(NamedTuple):
    recall_at_k: float
    mrr: float
    ndcg_at_k: float
    latency_p50_ms: float
    latency_p99_ms: float

def compute_recall_at_k(relevant_ids: list[set[str]], retrieved_ids: list[list[str]], k: int) -> float:
    recalls = [len(rel & set(ret[:k])) / len(rel) for rel, ret in zip(relevant_ids, retrieved_ids) if rel]
    return sum(recalls) / max(len(recalls), 1)

def compute_mrr(relevant_ids: list[set[str]], retrieved_ids: list[list[str]]) -> float:
    rrs = []
    for rel, ret in zip(relevant_ids, retrieved_ids):
        for rank, doc_id in enumerate(ret, start=1):
            if doc_id in rel:
                rrs.append(1.0 / rank)
                break
        else:
            rrs.append(0.0)
    return sum(rrs) / len(rrs)

Key metrics per stage:

• Retrieval: Recall@k, MRR, nDCG@k, latency p50/p99
• Reranking: nDCG@k on reranked list, latency per candidate
• Generation: Faithfulness, Answer Correctness, Context Relevancy (RAGAs framework)
• End-to-end: Token-level F1, EM score on QA benchmarks

Always evaluate on your domain. The table in Section 6.1 gives starting points, not guarantees.

Summary: What Actually Matters

1. Chunking is the foundation. Get it wrong and nothing downstream fixes it. Start with fixed-size recursive splitting (LangChain's RecursiveCharacterTextSplitter) with chunk_size=512, overlap=64. Upgrade to structure-aware or semantic chunking when you have evidence it helps.

2. Hybrid retrieval is the default. Pure dense misses exact matches. Pure BM25 misses paraphrases. RRF fuses them without score normalization headaches. Add SPLADE as a third leg only when paraphrase-heavy queries over technical text justify the cost.

3. Reranking is where precision lives. Bi-encoders cast a wide net; cross-encoders score with joint attention. The two-stage funnel (100 → 5) is the cost-conscious architecture. Load the reranker once at process startup — constructing it per query costs hundreds of milliseconds.

4. Query rewriting is cheap recall. Multi-query expansion and HyDE cost 1–2 LLM calls but often recover documents the raw query misses. HyDE turns query→document matching into document→document matching.

5. Metadata filtering is free precision. Hard constraints (date, category, product) applied at the vector DB level eliminate irrelevant candidates before they consume retrieval or reranking budget.

6. Parent-child retrieval solves the context gap. Retrieve small chunks for sharp signals, return large contexts for generation. Deduplicate parents, then apply MMR for diversity.

7. Measure each stage independently. Recall@k for retrieval, nDCG@k for reranking, faithfulness/correctness for generation. Optimizing one metric while ignoring others is how you get a system that retrieves perfectly but hallucinates in the answer.

Further Reading

• The LLM Stack Book, Chapter 9.4 — Chunking, Reranking & Hybrid Search (this article's primary source, with runnable Colab notebooks)
• Searching for Best Practices in RAG (Wang et al., 2024) — arXiv:2407.01219, systematic empirical study of RAG component combinations
• RAG Beyond the Demo (Xinyang Wu) — Pipeline, citations, evaluation, and when not to bother
• ColBERT / ColPali — Late interaction: the architectural middle ground between bi-encoders and cross-encoders
• GraphRAG / Agentic RAG — When retrieval itself becomes multi-step reasoning

If you found this useful, follow for more deep dives into the mechanics developers often misunderstand. Next up: how vector databases actually scale ANN search, or what model quantization actually does to your weights.