Recall@1 went from 13% to 50%. Recall@5 from 13% to 80%.
Over the same week, the number of questions my agent answered end-to-end went from 2 in 10, to 1 in 10, to 0 in 10.
Every component metric said I was winning. The product was dying.
The setup
A data analysis agent over a lakehouse of banking and economic time series. You ask a question in plain language, it finds the right series, builds a table, derives columns, runs change-point analysis, draws the chart.
The agent never sees the data. It sees a catalog, one row per measurable quantity, with a name, unit, frequency, coverage. It picks; the database computes.
The catalog has 55,532 entries. That number is the whole story.
The paradox
Semantic search had been off during a data migration. Retrieval was keyword matching, and keyword matching wants every token of your question to appear in the label. Ask for "inflation" and you get a balance-sheet adjustment line, because the consumer price index is called "Consumer Price Index" and the word "inflation" is nowhere in it.
before: recall@1 = 13% recall@5 = 13%
after: recall@1 = 50% recall@5 = 80%
Then I ran the end-to-end scenario. Before the retrieval work, 2 runs in 10 completed. After: 1 in 10. I assumed noise, improved retrieval once more, ran again: 0 in 10.
What the model was actually doing
Every failing run looked the same. Search the catalog. Search again. Twelve, seventeen, twenty-three searches in one turn — and never once call the tool that builds the table.
The obvious reading is indecision: at 55,532 options it cannot commit. I had three theories along those lines and started building a fix for the second one.
Then I turned on the reasoning trace and read what the model said to itself.
"The first search returned 8 candidates, but the output was empty (only '8 candidates, best to worst:' with no list). Let me try searching again with different terms."
"Strange. The search results are truncated."
It was not refusing to commit. It was looking for evidence that had been deleted out from under it.
The line
Context management had a rule. Keep the last three tool results verbatim, collapse older ones to their first line:
VERBATIM_RESULTS = 3
def _collapse(result: str) -> str:
return result.split("\n")[0]
Here is what a search result looks like:
8 candidates, best to worst:
IMF - GDP, Nominal (USD) · Türkiye (TUR) … [EVDS_TP.IMFGDPUSDN.TUR]
Consumer Price Index (General) … [EVDS_TP.FE.OKTG01]
… six more lines
The first line is the header. The rule kept the header and deleted the list.
The model was shown "8 candidates, best to worst:" with nothing after the colon, concluded the search had returned nothing, and searched again.
At ten searches in one turn:
total prompt 8,646 chars (~2,161 tokens)
tool results 30% of the window
candidate ids visible 18 of the 80 it had been shown (78% deleted)
The window was 30% full. The rule was destroying the turn's working memory to save 1,800 characters.
Why better retrieval made it worse
Better retrieval meant plausible candidates on the first search. Plausible candidates invite a second search to compare against. Every search past the third deleted another result.
- Bad retrieval → the model gives up early → few searches → few deletions
- Good retrieval → the model explores → many searches → the good candidates are destroyed
Improving one component degraded the system through a mechanism neither component owned. Recall@1 was honestly improving the entire time it was killing the product.
It turned out I had written the same mistake in two more places, an 800-character summary cap that silently cut lists at the fifth candidate, and a shortlist that filled oldest-first so it dropped whatever the newest search had just found. All three were correct for the catalog I had when I wrote them: thirteen entries. None were revisited when it grew four thousand times.
What fixed it
Not a smarter model, not a better prompt. Three changes about what the model is shown:
- a collapsed result keeps its content, not its label
- every search re-states the turn's accumulated shortlist, newest first
- results are grouped by catalog structure instead of returned as a flat ranked list
before after
runs completing all three turns 0/10 9/10
turns building a table ~7/30 29/30
discovery calls per turn 12 2
Retrieval quality did not change across that fix. Recall stayed at 50%/80%. Grouping changes presentation, not ranking — and presentation was the binding constraint all along.
"GDP, Nominal · Türkiye — one of 196 rows under this heading" is a decidable choice. The same line in a flat list is not.
The industry-standard fix that backfired
Faced with "the model doesn't know how a question of this shape is answered here", the textbook answer is a verified query repository: question-and-answer pairs injected as worked examples. Snowflake ships exactly this. I built it.
grouped results only 9/10 runs
+ verified query repository 3/10, then 5/10
It cost six runs in ten. Half the damage had a cause written in my own interface contract six weeks earlier:
The system prompt must not contain family lists or series ids. A model that has seen an id in its prompt will invent variations on it.
My examples carried the ids of the families they resolved to. I had read that file. I shipped it anyway.
The other half was simpler: with grouped results, the model could already see where matches clustered. An abstract example gave it something extra to reason about that it did not need. Snowflake needs the repository because its semantic model is capped at 32K tokens and it cannot show the catalog. I could. Two solutions to one problem, interfering.
Three things I'd tell myself a week earlier
Read what the model says to itself. Three sessions of theorising about indecision were settled by one run with the reasoning trace on. The model had been describing the bug in plain language the whole time.
Component metrics can rise while the system dies. Recall went up monotonically across the exact window where end-to-end went to zero.
Every constant has an invisible scale attached. VERBATIM_RESULTS = 3, an 800-char cap, oldest-first fill — all correct at thirteen entries, all silently wrong at 55,532. Grep your constants and ask what size they were written for.
Top comments (9)
The
_collapsereturningresult.split("\n")[0]detail is the whole article, and it's brutal because it's the kind of rule that's completely correct in isolation. The header line "8 candidates, best to worst:" survives, the actual candidates get truncated, and the model's reasoning trace ("the output was empty... let me search again") is it behaving rationally on corrupted evidence. Two things stuck with me. First, you only caught it by reading the reasoning trace â component metrics (Recall@k) literally can't see it, because retrieval genuinely did get better; the damage happened downstream in context assembly. We've started treating "searches-per-turn" as a first-class health signal for exactly this reason: a spike means the agent is re-fetching evidence it should already hold. Second, a 55k-entry catalog punishes any lossy compression of tool results far harder than a small one would. Did you end up making the collapse content-aware (keep the candidate list, drop only prose), or did you just bump VERBATIM_RESULTS and eat the token cost?Content-aware, and the token cost turned out not to be the trade-off I expected.
Bumping VERBATIM_RESULTS was the first thing I tried and it doesn't actually fix the mechanism. It buys you three more searches before the same deletion starts, and with 55k entries the model will happily use them. The bound just moves.
What landed:
1- a collapsed result keeps what identifies its content, not its label. For a candidate list that's names and ids, compacted; for an analysis result it's the finding, not "2 findings"
2- every new search re-states the turn's accumulated shortlist inside its own result, newest first. The newest result is the one the collapse rule always keeps, so the shortlist rides along on the one thing that can't be truncated.
That second one is the part I'd keep in any agent. It makes the turn's working memory self-healing rather than dependent on a retention policy being right.
And I got it wrong once on the way: the shortlist hit the 800-char summary cap and filled oldest-first, so past the cap it kept the earliest finds and dropped whatever the newest search had just produced. I'd recreated the original bug one layer in. Only the end-to-end number caught it. 18/30 turns down to 8/30.
On tokens: the window was 30% full at ten searches. The rule was destroying the turn's working memory to save about 1,800 characters. There was never a token problem to solve; I'd just assumed there was, because the rule was written when the catalog had thirteen entries.
Your searches-per-turn signal is the right instrument and I wish I'd had it. Ours went from a median of 12 to 2 across this fix. And that number moved while recall stayed flat at 50/80, which is exactly the decoupling you're describing. I'd add one more: discovery calls that return candidates the turn has already seen. A repeat is the specific symptom of evidence loss, where a high count alone could just be a hard question.
This is the cleanest proof I’ve seen that a rising component metric can kill the product: recall climbed while completions went 2/10 → 0/10 because context assembly deleted the evidence the better retrieval had just found.
The eval contract I’d pin next to Recall@k is end-to-end completion plus a context-health cut (searches-per-turn, and re-fetches of candidates the turn already saw). Those move when the binder is broken even if retrieval stays flat.
Component win ≠ system win until the evidence the retriever returned still exists when the agent decides.
Component win ≠ system win until the evidence the retriever returned still exists when the agent decides".That's a tighter framing than anything in the post. Stealing it.
The re-fetch signal is the one I'd push hardest. Searches-per-turn alone is ambiguous: a hard question legitimately costs more searches. A repeat is not ambiguous, the agent asking again for something it was already shown means the binder dropped it, and there's no benign reading.
One thing I'd add to the contract from how this actually failed: the eval has to move with the data, not just the code. Ours was written against a thirteen-entry catalog and stayed valid while the catalog grew four thousand times. Every constant in the binder; retention count, summary cap, fill order; was correct when written and silently wrong afterwards, and nothing in CI knew a scale had changed underneath it. The end-to-end number is the only thing that noticed, and it noticed late
Twelve searches down to two is the number worth pulling on. The way I read it, grouping landed the model on a decision frame in one shot where a flat list leaves it with a fragment and a reason to look again. I ran into something close in an indexing pipeline where the fix wasn't better ranking. It was narrowing the candidate set to the point where the model would actually commit.
Same conclusion, opposite direction. Narrowing measured worse for me. I tried widening the pool and diversifying the members, and runs stopped completing at all; the extra candidates ate the step budget. What moved it was making the frame legible, not smaller: which headings the matches cluster under and how many siblings each holds. "Türkiye (TUR) GDP, Nominal — one of 196 rows under this heading" is decidable. The same line in a flat list isn't, and reformulating is the honest response to an undecidable list.
The 55k catalog makes this painfully clear. Retrieval can improve while the agent regresses when context assembly hides the evidence.
Scale is the part I underestimated. At thirteen entries the same collapse rule was harmless, re-searching was cheap and a candidate list fit in the cap anyway. At 55k, every deleted line is a candidate the model won't find again, and it pays for that by searching more, which deletes more. The rule didn't change; its meaning did.
Recall climbing while end-to-end results drop is a good warning about offline metrics. Better top-1 hits can still hand the model three near-miss passages that it blends into one wrong answer. Judging each retrieval change by whether the task actually got resolved, and retiring the one that only won the offline metric, is the check I'd lean on.