Short answer: give retrieval a measured latency budget, then spend the biggest slice on the index rather than trying to make the language model hide a slow search. For a media team's internal knowledge-base bot, I start with a 120 ms retrieval target, keep a lexical fallback, and choose the smallest candidate set that passes answer-quality tests.
That recommendation is about controlling index cost at scale. A course tutor and a newsroom bot ask different questions, but they share the same failure: a semantic query that feels clever in a notebook becomes an expensive queue in production. The fix is an explicit contract between embedding, index lookup, reranking, and answer generation.
The experiment: where did the milliseconds go?
I begin with a trace, not a vector database. The trace records query normalization, embedding time, first-stage search, optional reranking, and time-to-first-token. Every span carries the corpus version and the number of candidates. Without those fields, a p95 regression looks like “the model got slower” when the real change was a rebuild with twice as many chunks.
For an online course tutor, a reasonable first contract is 300 ms to show a useful answer, with 120 ms reserved for retrieval. The remaining time covers application work and generation. Those are planning numbers, not facts about every deployment; your mileage may vary with network distance, hardware, and model size.
The simple approach is top-k cosine search over every chunk, followed by a large prompt. It is easy to demo and hard to budget. At 50,000 chunks, increasing k from 8 to 40 also increases serialization, reranker work, and prompt tokens. The answer may improve on a small test set while the experience gets worse for every learner.
My experiment therefore compares three knobs separately: chunk count, candidate count, and reranking depth. I hold the answer model and evaluation set constant. A useful result is not “semantic search won”; it is “k=12 keeps recall above the acceptance line while p95 stays below 120 ms.” In one representative replay, a transcript update doubled the number of chunks because a splitter treated every caption timestamp as a new paragraph. The dashboard showed a 14 ms index increase, which looked harmless until the reranker began receiving 40 near-identical candidates and the prompt grew past its normal token range. I traced the request by corpus revision, inspected the candidate IDs, and found that the duplicate timestamps had survived normalization. After collapsing adjacent caption lines and lowering k from 24 to 12, the first-stage hit rate stayed flat on the reviewed questions while reranking work fell sharply. That is the sort of result I trust: a named data change, a visible mechanism, and an evaluation check that can be rerun when the archive changes again. If the same fix only improves a single hand-picked question, it is not a production decision yet.
What should a semantic retrieval latency budget include?
Treat the budget as a sum of observable stages:
| Stage | Starting target | What to record |
|---|---|---|
| Query cleanup and routing | 10 ms | language, filter count |
| Embedding | 45 ms | model, input tokens |
| First-stage index search | 40 ms | index type, k, shard |
| Reranking | 25 ms | candidates, model, cutoff |
The total is 120 ms, but the table is not a promise. It is a debugging map. If embedding consumes 90 ms, shaving 5 ms from filtering will not rescue the request. If search is fast but reranking dominates, lower the candidate set before changing infrastructure.
There is also a queue budget. A service that answers in 80 ms when idle can miss its target under ten concurrent requests. Track p50, p95, and p99 separately, plus queue wait. Averages are polite liars here.
For a media knowledge base, filter before vector search when the question names a show, date range, or access group. Metadata filters reduce both distance calculations and accidental leakage. Keep the filter expression in the trace so an empty result can be distinguished from a genuinely unfamiliar question.
A small Python harness catches expensive intuition
The harness below does not depend on a hosted product. It treats the retriever as a callable and evaluates latency and recall together. The gold set can come from support tickets or tutor lesson checks, with references reviewed by a subject-matter owner.
from dataclasses import dataclass
from statistics import quantiles
from time import perf_counter
from typing import Callable, Sequence
@dataclass
class Case:
question: str
expected_ids: set[str]
def evaluate(
retrieve: Callable[[str, int], Sequence[str]],
cases: Sequence[Case],
k: int,
) -> dict[str, float]:
elapsed_ms: list[float] = []
hits = 0
for case in cases:
started = perf_counter()
found = set(retrieve(case.question, k))
elapsed_ms.append((perf_counter() - started) * 1000)
hits += bool(found & case.expected_ids)
if not elapsed_ms:
raise ValueError("evaluation set must contain at least one case")
p95 = quantiles(elapsed_ms, n=100, method="inclusive")[94]
return {
"hit_rate": hits / len(cases),
"p50_ms": quantiles(elapsed_ms, n=2, method="inclusive")[0],
"p95_ms": p95,
"max_ms": max(elapsed_ms),
}
The important detail is the paired result. A candidate configuration must clear both a quality threshold and a latency threshold. For example, reject a configuration if hit rate is below 0.85 or p95 is above 120 ms. I would rather ship a lexical result with a clear “no matching note found” state than silently stuff low-confidence chunks into the prompt.
There is a trap in tiny evaluation sets: one slow cold-start request can move p95 dramatically. Keep warm-up runs separate, report the sample size, and replay queries from each important content type. Transcripts, caption files, and policy pages have different chunk lengths and filtering patterns.
Why chunking and index choice change cost together
Chunking is an index decision, not just a preprocessing detail. Splitting a transcript into 200-token windows creates more vectors and often more near-duplicates. Splitting it into 800-token windows reduces vector count but makes each hit less precise and increases the tokens sent downstream. The right boundary follows the unit a human would cite: a scene, a lesson objective, or a policy clause.
I keep the original document ID, section heading, character offsets, and a content revision on every chunk. That metadata makes re-indexing incremental. When an editor changes one paragraph, the pipeline can replace the affected chunks instead of embedding the whole archive again. Hash the normalized text and skip unchanged hashes; this is one of the few cost controls that does not require a new search engine.
Approximate nearest-neighbor indexes trade exactness for speed. Their parameters should be tuned against the gold set, not copied from a benchmark with a different vector dimension or corpus shape. Increasing graph connectivity or search effort can lift recall while raising build and query cost. Quantization can lower memory pressure, but measure the recall loss on short questions and named entities before adopting it.
A two-stage layout is usually easier to reason about: cheap lexical or metadata filtering narrows the universe, semantic search retrieves a modest candidate set, and a reranker sees only those candidates. The stages have separate timeouts. If reranking reaches its deadline, return the first-stage order with a lower confidence label. The system remains useful and, more importantly, observable.
How can a course tutor avoid semantic retrieval failure modes?
Start with the failures learners actually report. “The tutor cited an old grading rule” is a freshness and filtering bug, not proof that embeddings are bad. “It answered a neighboring concept” points to chunk boundaries or an overly generous k. “It paused for a second after every follow-up” points to repeated embedding and cache misses.
Use a query cache only for normalized, permission-safe inputs. Cache the embedding separately from retrieved IDs, because content revisions invalidate IDs while the query vector may remain useful. Include tenant, locale, and corpus revision in the cache key. Never let a cache bypass authorization checks.
For follow-up questions, carry a compact resolved subject (“lesson 4, binary trees”) instead of the entire chat transcript. This lowers embedding input tokens and makes the query trace explainable. If subject resolution is uncertain, ask a clarification question; guessing creates a retrieval miss that looks like a generation problem later.
The catch is that this architecture is not suitable when answers must reflect continuously changing records with strict transactional semantics. A relational query or a domain API should own that truth, with retrieval used for explanatory context. Stick with a direct structured lookup when the user asks for an exact count, current entitlement, or irreversible action.
Measure before copying the 120 ms number
Before adopting any budget, build a small matrix: corpus size, concurrent requests, cold and warm states, filter selectivity, and query language. Run it against a fixed evaluation set and save the trace distribution with the index artifact. A budget that passes on 10,000 chunks may fail at 10 million; an index that is cheap to query may be costly to rebuild every hour.
I also review misses by category, not just one score. Recall@k, citation correctness, stale-content rate, empty-result rate, and token count tell different stories. Set a rollback rule for each index release. If p95 crosses the limit or stale citations rise, restore the previous artifact while the new one is investigated.
The research framing for retrieval-augmented generation is useful here: retrieval supplies evidence, while generation conditions on that evidence. It does not remove the need for access control, freshness policy, or latency accounting. Those remain application responsibilities.
Three words: trace every stage.
That habit keeps notebook experiments honest. It also gives a media team a defensible answer when someone asks why a semantic retrieval change raised the bill or made a tutor feel slow. The best index is the one whose quality, cost, and failure behavior you can explain before you ship it.
Top comments (0)