I built a batch question-answer extractor because I was tired of watching agents loop one question at a time through an LLM, paying latency and tokens for every single call. The idea was simple: give it a document and 1,000 questions, and get back structured answers in a single pass without the model generating a single word of prose.
The concrete trick is that it never asks the model to write answers. Instead, it embeds the document once, embeds all 1,000 questions in one batched request, and does a single vector similarity sweep to map each question to the most relevant passage. The "answer" is just the source span — no generation, no hallucination surface, no token-by-token streaming. The whole thing resolves in about 200 milliseconds because the heavy lifting is one embedding pass plus a matrix multiply, not a thousand autoregressive decodes.
The thing it costs the operator is memory headroom and embedding throughput. A 1,000-question batch with a long context document means holding a sizable embedding matrix in RAM, and if your questions exceed the batch window you have to chunk carefully or you silently drop coverage. I hit that ceiling during testing and had to add a sliding window that trades a little accuracy for predictability.
I recorded the full walkthrough — the architecture, the chunking edge case, and the latency numbers — and put it up here: https://www.youtube.com/shorts/M6voMv_ubIw
Top comments (0)