At the end of part 2 I promised a clean competitor comparison: more than one memory system, the same admission gate, enough capacity to finish.
This is that run. 1,453 agent sessions, four arms, five corpus conditions, every outcome graded by executing a checker against the code the agent actually wrote. No LLM judge anywhere.
The corpus
Each condition is a 4,911 document corpus, about 27,000 embedded chunks. Of those documents, 4,902 are adversarial distractors generated from the corpus's own vocabulary to be confusable, and the rest are the planted material for 36 tasks.
That ratio is the point. On the small feed I used earlier, retrieval hit@10 was 1.000: everything was findable, so the benchmark could not tell "the product retrieved badly" from "the agent never searched". With the haystack, BM25 hit@1 drops from 0.485 to 0.182.
The five conditions differ in what is planted, and four of the five are traps:
| condition | what the corpus holds | share of cells |
|---|---|---|
present |
the governing fact, cleanly | 37% |
superseded |
the fact, beside an outdated version of itself | 15% |
adjacent |
something related and inapplicable | 16% |
absent |
the fact deleted, distractors left behind | 17% |
contradictory |
two rival versions, no authority between them | 15% |
Only 37 percent of cells contain a clean fact worth retrieving. In the other 63 percent the corpus is outdated, contradictory, inapplicable, or empty, and a memory layer's job there is to not make things worse.
I built it that way deliberately. A benchmark where memory can only help is a demo.
The control that changed the question
Two commenters asked for a placebo. Fair: if you tell an agent "you have memory, consult it before acting" and it then does better, you have not separated the memory from the telling.
So I built an arm carrying the complete memory instruction and no memory at all. Same protocol text, byte for byte, as the two real arms. Nothing behind it.
| comparison | wins | losses | net | p |
|---|---|---|---|---|
| instruction only, vs bare | 26 | 43 | -17 | 0.053 |
Telling an agent to consult memory, when there is no memory, makes it measurably worse. Seventeen cells of 358, borderline significance, in the direction nobody advertises.
That reframes everything. "Does memory beat no memory" is not directly answerable, because every memory product requires that instruction. The answerable question is: does the product earn back the cost of asking for it?
The comparison
RE-call 0.11.0 is mine, so read accordingly. MemPalace 3.8.0 is the better known of the two. Both received the same instruction and the same gate. Every write tool was withheld from every arm.
| comparison | wins | losses | net | p |
|---|---|---|---|---|
| RE-call vs instruction only | 41 | 21 | +20 | 0.015 |
| MemPalace vs instruction only | 25 | 23 | +2 | 0.885 |
| RE-call vs MemPalace | 48 | 30 | +18 | 0.054 |
| RE-call vs bare | 47 | 44 | +3 | 0.834 |
| MemPalace vs bare | 38 | 53 | -15 | 0.142 |
McNemar exact on discordant pairs. The only comparison clearing p < 0.05 is RE-call against the instruction-only control.
Note the fourth row before anyone else does. Against a bare agent, RE-call is +3 cells at p = 0.834, which is nothing. The win is real against the instruction the product requires and invisible against an agent you never asked. A sceptic will quote that row and is entitled to.
Arm by arm
Tasks solved out of cells, admitted only where all four arms produced a record. Ordered by how much RE-call gains over the instruction-only arm, which is the comparison that isolates the product from the coaching.
| condition | cells | bare | instruction | RE-call | MemPalace | RE-call gain |
|---|---|---|---|---|---|---|
| present | 134 | 52/134 · 0.388 | 53/134 · 0.396 | 67/134 · 0.500 | 56/134 · 0.418 | +14 |
| superseded | 52 | 38/52 · 0.731 | 34/52 · 0.654 | 41/52 · 0.788 | 35/52 · 0.673 | +7 |
| absent | 60 | 49/60 · 0.817 | 41/60 · 0.683 | 44/60 · 0.733 | 37/60 · 0.617 | +3 |
| contradictory | 54 | 44/54 · 0.815 | 41/54 · 0.759 | 39/54 · 0.722 | 42/54 · 0.778 | -2 |
| adjacent | 58 | 42/58 · 0.724 | 39/58 · 0.672 | 37/58 · 0.638 | 40/58 · 0.690 | -2 |
| all | 358 | 225/358 · 0.628 | 208/358 · 0.581 | 228/358 · 0.637 | 210/358 · 0.587 | +20 |
RE-call is the only arm that beats doing nothing, by 0.9 points overall. MemPalace at 210/358 sits below the no-memory baseline and barely above instruction-only. MemPalace's gains against the same control are +3, +1, -4, +1, +1, totalling +2.
The shape of that table matters more than the total. Memory pays where there is something to recall and costs where there is not. On present RE-call converts 67 of 134 against bare's 52. On the three conditions built to punish retrieval it is flat or slightly negative, because surfacing something inapplicable is worse than surfacing nothing.
This is close to a worst case, and that is the point
Go back to the composition. Sixty-three percent of these cells are conditions built to punish a memory layer, where the corpus is outdated, contradictory, inapplicable, or empty and the best available outcome is to waste nothing. Only 37 percent contain a clean fact worth retrieving.
RE-call is still the only arm that finishes above the no-memory baseline. It does that while spending nearly two thirds of its cells in terrain designed to make retrieval a liability, and while paying the -17 cell entry fee that the instruction alone costs.
That is the number I would defend hardest, because it is the unflattering way to measure my own product and it still comes out positive.
And the composition is a choice I made, not a fact about anyone's repository. Real memory does not sit still at that ratio. How much of it is current, governing and correctly superseded is a property of how the layer writes and maintains notes over time, which this benchmark does not measure at all. It fixes the corpus and varies the product.
So read the table as a floor rather than a forecast. On the condition a well maintained corpus is mostly made of, RE-call is +14 of its +20. What that becomes at a healthier ratio I have not measured, and I am not going to put a number on it.
I would rather someone else did. A composition sweep, holding the product constant and varying the mix, is the obvious next experiment and it is not run.
The number I find most telling
For each product, the solved rate when it searched minus when it did not:
| condition | RE-call | MemPalace |
|---|---|---|
| present | +0.181 | +0.019 |
| adjacent | +0.147 | -0.032 |
| contradictory | +0.079 | +0.085 |
| absent | +0.031 | +0.264 |
| superseded | +0.006 | -0.094 |
When RE-call searches it helps in five conditions of five. When MemPalace searches it hurts in two, while searching more often (0.58 to 0.69 against 0.46 to 0.56).
On present, the condition built so the answer is in memory, searching is worth +0.181 to RE-call and +0.019 to MemPalace. That is the cell both products exist for.
Cost, median beside mean because these distributions are skewed enough that on an earlier run of mine the two disagreed in sign: RE-call 65.9 s and 56,476 input tokens per session, MemPalace 68.6 s and 90,706, bare 44.7 s and 14,085. RE-call is faster and uses 38 percent fewer tokens than MemPalace while delivering ten times the net benefit. Both are materially heavier than no memory, which is the honest cost of the idea.
What I had to publish against myself
The previous run said something different, and I want to be the one who says why.
In it, RE-call looked like the most damaging arm of seven, worst on three of four adversarial conditions. I had written that up as a genuine weakness of my own product.
It was my harness. That run gave RE-call 1,958 bytes of coaching over the shared protocol against MemPalace's 853, and most of the surplus was not about RE-call at all: search before your first write, search by symptom rather than goal. Generic advice that helps any retrieval system, handed to one of them. Worse, the flag selecting the fair variant was hardcoded in my launcher with no override, so no run I had ever done could have been matched, and I had not noticed.
Under the corrected instruction the finding evaporates. RE-call's damage-only net harm falls from +0.073 to +0.0122, identical to MemPalace's, and its superseded damage from 0.179 to 0.036. Its usefulness score also fell, 0.377 to 0.281, while MemPalace's did not move at all, because MemPalace had been on the shared protocol all along. My product's advantage narrowed from 1.8x to 1.35x once I stopped giving it a bigger prompt.
Both corrections point the same way. I had been generous to myself and unfair to the competitor, and it produced one flattering finding and one damning one, both wrong.
What this benchmark cannot see
RE-call has a reasoning-graph layer. Across 2,181 sessions of the earlier run its four graph tools were called zero times. Read alone, damning.
It builds that graph from authored cross-references, the way a wiki links notes. Its own memory store has 44,364 of them. This benchmark's corpus is rendered agent transcripts: conversation turns, no cross-references. The graph gets entities and no edges. It is a node set, not a graph.
So graph reasoning needs an authored, cross-referenced corpus and this benchmark deliberately measures raw transcripts. Different corpus classes, not a weakness. Manufacturing cross-references in to make the feature look good would be assuming the answer. I established that with ten read-only queries rather than the 220 sessions the experiment would have cost, after a reviewer pointed out my draft design would have measured a language model reading trusted evidence and published it as a graph result.
Come and break it
Two systems is a comparison, not a field, and one of them is mine. The harness is public: github.com/GiulioDER/agent-memory-bench.
If you build a memory layer for coding agents, I would rather measure yours than have you take my word for this. Every adapter is hash-pinned and every vendor is invited to review their own config before a measured run, which is what adapters/<name>/VENDOR_REVIEW.md is for. Open an issue and I will help wire it up.
And if you just want to check my arithmetic, the README has a section on verifying a published run with no credentials and no money. The preregistrations are in the repo, including the ones whose predictions I got wrong, and the raw records are published with each run.
If you take one number from this, take that one: 63 percent of the cells were built to make memory a liability, and the memory layer still finished ahead of not having one.
One thing I would most like broken: the 45 percent. On superseded, the largest single loss is that in 45 percent of sessions neither planted document is retrieved at all. That is worth roughly three times more than anything I have measured on the ranking side, it is a query-formulation problem rather than a retrieval-quality one, and it is already preregistered. If someone has a better idea than mine, I would like to lose that one.
The method, the leaderboard and the submission rules now live at giulioder.github.io/agent-memory-bench
Top comments (9)
The placebo arm is doing the heaviest lifting in this design, and it is one arm where two different controls are possible, with different meanings. An instruction that points at tools which do not exist makes the agent burn turns on failing calls, and that is a cost no user of a real product ever pays. An instruction over a real store that is simply empty returns a clean no-results and moves on, and that is the actual entry fee of any first session. The -17 bounds different things depending on which one was built. If it was the tool-absent kind, the entry fee is overstated, and the +20 of the full system quietly inherits the overstatement, because earn-back is measured against the wrong baseline. The reframe from does memory help to does it earn back the cost of asking is the right question, which is exactly why the cost side deserves the same rigor the benefit side already got: name which placebo this was, and if it was the first kind, the empty-store arm is one more run and it would make the earn-back number hard to argue with. The executing-checker grading and the trap density already put this series ahead of most published evals on the benefit side, so the one ambiguous control stands out more than it would elsewhere.
One aside, apart from the method. The covers changed in late August, and for the better. The header is quieter, a single word carries the accent, and the numbers have room of their own. Measurement-first writing converges on this grammar sooner or later, and I take the convergence as a good sign for where the genre is heading.
That is a fair distinction, and I can clarify the implementation. The arm labelled “instruction only” was not given nonexistent memory tools. It kept the normal repository tools, and its instruction directed it to search the repository with Grep and Read. It had no memory store or memory tools, so it did not pay a failed memory call penalty.
It was therefore a no memory control, but not the same thing as an empty real store that exposes the memory interface and returns no results. The −17 estimates the cost of this particular control and instruction, not a universal entry fee for every memory product. Likewise, the +20 is relative to that baseline, not a claim that RE-call earns back its cost under every deployment.
I should have named that distinction in the post.
One further qualification matters here: this is an adversarial benchmark, not a typical first session. The benchmark is designed to punish indiscriminate retrieval.
So the −17 is a stress test under a particularly hostile corpus mix, not a claim about the normal entry cost of a well maintained real store. An empty real store would still be a useful follow up, because it would separate interface overhead from the difficulty created by the adversarial conditions.
That clarification does name it, and it is a cleaner control than either of my two kinds: no interface at all, with the instruction spending the budget on repository search instead. It also sharpens what the -17 is. Under a corpus built to punish indiscriminate retrieval, an instruction that says consult something before you act taxes the agent with extra reads exactly where extra reads are the trap, so the arm prices obedience under hostile conditions rather than interface overhead. Which is why the empty-store follow up completes the design rather than padding it: presence of the interface and target of the instruction are two variables, and the current arms move both at once. The empty store holds the instruction constant and isolates what the interface itself costs, and my guess is it lands well under 17, which would make the earn-back story stronger for any product that clears it, yours included. The caveat you added is the part I would promote into the post body: a stress test under adversarial mix is the right reading of -17, and readers will otherwise quote it as the universal entry fee, because a single number always escapes its conditions.
Yes, and that is the broader scope of the benchmark. Retrieval quality is a valid measure if retrieval itself is the product claim. For a coding agent retrieval is only useful if the agent also knows when to search, how to interpret what it finds, and when not to trust it.
A memory layer can retrieve a relevant passage and still make the agent worse if that passage is absent, superseded, contradictory, or unrelated to the task. That is why I included adversarial conditions. They are deliberately hostile, but they test the property that matters for an agent: whether retrieval improves completed work rather than merely producing plausible search results.
In that sense, this benchmark starts where a pure retrieval benchmark stops. High recall can show that a fact is findable. It cannot show that the layer is useful in a coding workflow, or that it avoids negative transfer. Without adversarial testing, a high retrieval score can become a marketing claim rather than evidence of practical usefulness.
Six pairwise comparisons on the same 358 cells, and the one that lands does so at p = 0.015, which sits above the Bonferroni line for a family that size: 0.05/6 is 0.0083, and 0.05/5 is 0.010 even setting the placebo row aside. The per-condition table adds five more gains on top of that. So whether 0.015 is a result or a maximum over the family turns on whether RE-call against instruction-only was named the primary endpoint before the run, and the post does not say either way.
It is worth saying because you already flagged the row that makes it decisive. Instruction-only is -17 against bare, so the single significant comparison is against an arm you demonstrated is worse than doing nothing. If that endpoint was fixed in advance, that is the strongest sentence available to you here, and it is not in the post.
The multiplicity point is fair, and I should have made the status of that p value clearer.
RE-call versus instruction-only was the intended practical contrast, because it asks whether the memory layer earns back the cost of the instruction it requires. However, it was not registered as a primary endpoint before this run, so p = 0.015 should be read as an unadjusted exploratory result, not familywise confirmatory evidence.
I still think the comparison is informative, especially alongside the −17 result for instruction-only versus bare. But I agree that this context does not remove the multiple comparisons issue. Against bare, RE-call was only +3 cells, with p = 0.834, so the careful interpretation is a gain over the instruction-only cost, not a general improvement over doing nothing.
I’ll clarify that distinction and add the preregistration status to the post. A confirmatory follow-up should predeclare the primary contrast or adjust across the full comparison family.
The dangling-case diagnostic appearing once and in no test is the "guard that has never fired" class - the resolver does the right thing today, and nothing would notice the day it stops. A twin test that plants exactly that dangle belongs in your 222-case known-bad set; it would be the cheapest test in the file.
The 45 percent holding as the dominant class settles where the work is. So, formally: we would like to be measured - adapter request is open at github.com/GiulioDER/agent-memory-..., with the version pinned and a bulk-ingest path offered so the corpus loads do not fight our rate limit. Two commitments stated before any number exists. First, we will clone the harness and run privately against our own adapter before any published run - the public run should not be our rehearsal. Second, a pre-registration against ourselves: we expect your superseded condition to hurt us, because our store currently lets successes overwrite while corrections merely append - a defect we documented against ourselves last week, with the fix still on the backlog. If your harness confirms it, you will have measured our backlog, and that is worth more to us than a flattering row.
Three days on, and the pre-registration from my comment above is settled. It went the way we said it would.
We cloned the harness and ran all four conditions privately, three times over - 005c, 006, 007, 120 cells each. On the 11 planted superseded cells in the last run, our memory arm solves 8 and the memory-less arm solves 9. We are one behind an agent that was given nothing, on the condition we named in advance as our weak spot.
One run per cell settles nothing on its own, and I would rather say that than lean on it: an earlier run had us at 10 of 11 there. What survives across both is the direction - superseded is where our margin over bare is thinnest, and it is the one we called.
So we went and measured the mechanism in our own store instead of inferring it from your cells. It is worse than we described in the request: 735 stored entries, two supersession edges - and those two point at each other. About 12 % of entries contain an explicit correction word ("instead", "no longer", "turned out"); none of that 12 % was linked to what it corrected. The store had the evidence that it was contradicting itself and nowhere to write it down.
On the 45 % you said you would most like to lose, we may have half of something, and it is not about query formulation. Changing one value - the recall renders ten results instead of five, nothing else - moved superseded from +4 to +10 against bare, with fewer searches per session (3.10 → 2.60) and 4 % fewer tokens. If in 45 % of sessions neither planted document surfaces, a wider window is the cheapest thing to try before rewriting queries. It is one config value and no new component. It is also not free: in the conditions with nothing to compare it costs 8–13 % more tokens for nothing, and it misses two of our own pre-registered cost floors.
One thing for the thread rather than for us. Because a restart made us run present three times with identical inputs, we can put a number on repeat variance: the memory-less arm alone moved by four cells of thirty. The difference to bare in the same run moved by one. That is an unglamorous companion to the p-value discussion above - at one run per cell, a per-condition ±2 is not a finding in either direction, including ours.
All of it, including the parts that do not flatter us and the four ways our wiring differed from your checked-in config, is in PR #78 - reports/vendor-cachly-005c-007.md and the filled-in VENDOR_REVIEW.md. Run 008 is finished and will be published the same way.
I read 005c, 006, and 007 as private rehearsals, not as runs of the benchmark, and I agree that the four wiring differences are material. The shipped corpus, bare baseline, reduced tool set, and altered instruction make those results useful diagnostics, but not official evidence about the checked in configuration.
Your planted versus untouched split and repeat variance are valuable findings. I will treat them as narrower observations about the saturated feed until the same analysis is run with the generated haystack, claude_md, all six read tools, and the shipped instruction.
I checked the empty answer question. A blank final response is not counted as an abstention. Abstention requires an explicit recognized decline or conflict marker. absent and contradictory are the abstainable conditions. On superseded and adjacent, declining is a miss because the corpus contains an applicable answer.
I also checked the ts-round-money concern. The committed oracle expects 5.67, and the supplied oracle inputs produce 5.67 with per line ROUND_HALF_UP. I cannot reproduce 18.05 from the benchmark fixture. If that value comes from a particular transcript, please share the exact record so I can trace it.
The current gate treats session errors as inadmissible and excludes discarded cells from endpoint calculations. I cannot independently rederive the private tables from this PR because the raw session records are not included. Once Run 008 is published, I will check it against the same evidence boundary.