The first experiment measured whether Claude Code knew the right answer. This one measured whether the work actually succeeded.
In my previous experiment, I tested whether a memory layer could stop Claude Code from confidently repeating project-specific mistakes.
It did.
But there was an obvious limitation.
The tasks measured whether the answer was correct. They did not ask the agent to modify a repository, run the result, and pass a deterministic test.
So I ran the harder version.
This time, the agent had to do real work in a temporary repository. A checker decided whether the result was correct. No judge model. No subjective scoring. The task either passed or it did not.
Here is the result from the complete run:
| Configuration | Successful runs |
|---|---|
| Claude Code without memory | 50.0% |
Claude Code with CLAUDE.md
|
36.1% |
| Claude Code with RE-call | 58.3% |
RE-call improved success over the CLAUDE.md baseline by 22.2 percentage points.
The 95% confidence interval was +11.1 to +33.3 points.
The paired McNemar test gave a p-value of 0.000145.
This was not one lucky demonstration. It was 72 paired comparisons across 24 tasks and three seeds, with zero discarded cells.
What changed from the first experiment?
Each task asked Claude Code to create or modify files in a repository and run the result.
The benchmark included tasks involving things like:
- deterministic file ordering
- Unicode normalization
- migration naming
- shell quoting
- money rounding
- stable sorting
- line ending handling
- generated files and golden outputs
The important detail is that every task had a deterministic checker.
The agent could say "done" all it wanted. The checker only cared whether the repository ended up in the correct state.
I ran every task through three configurations:
- Bare Claude Code
- Claude Code with the project's
CLAUDE.md - Claude Code with the same environment plus RE-call memory
The harness also verified that the memory tools were actually available before counting a session. If the memory layer was missing, the entire paired cell was discarded.
That happened zero times in the DeepSeek run.
RE-call helped where project memory mattered
The most useful comparison is RE-call versus the static CLAUDE.md file.
The static file produced a 36.1% success rate.
RE-call produced 58.3%.
That is a 22.2 point improvement.
The paired results were even more revealing:
- RE-call won 17 comparisons that
CLAUDE.mdlost -
CLAUDE.mdwon only 1 comparison that RE-call lost
The effect was strongest on the tasks classified as memory-sensitive. Across those eight tasks, RE-call improved success by 45.8 percentage points.
That is the pattern I was looking for.
Memory did not make every task easier. It helped when the task depended on a project decision, convention, or failure mode that was not fully present in the immediate prompt.
The surprising result was CLAUDE.md
The static CLAUDE.md configuration performed worse than the bare configuration:
| Configuration | Successful runs |
|---|---|
| Bare | 50.0% |
CLAUDE.md |
36.1% |
The bare configuration won by 13.9 percentage points.
This is not what I expected.
It suggests that a static instruction file is not automatically a memory system. It can add useful guidance, but it can also add noise, constraints, or instructions that compete with the task.
RE-call is doing something different.
Instead of putting every historical fact into the permanent prompt, it gives the agent a way to retrieve relevant project history when the current task needs it.
That distinction matters.
Memory did not just improve the final score
The benchmark also measured what happened inside the memory mechanism.
In the DeepSeek run:
- The agent searched memory in 83.3% of eligible sessions.
- When it searched, it reached useful context 85.0% of the time.
- Overall, useful project context was reached in 70.8% of sessions.
So the result is not simply that "the model had more tokens."
The memory layer was being consulted, and in most cases where it was consulted, it reached something useful.
There is still room to improve retrieval. A search is not automatically a good search. The model needs to find the right historical decision, not merely retrieve something related to the words in the prompt.
The cost
The complete DeepSeek run cost an estimated $0.4964 at the captured API prices.
The three arms cost:
| Configuration | Estimated cost |
|---|---|
| Bare | $0.0824 |
CLAUDE.md |
$0.0863 |
| RE-call | $0.3277 |
RE-call used about four times as many total tokens as the static prompt configuration.
That is the tradeoff.
The memory layer costs more on every task, including tasks where memory is unnecessary. In exchange, it can prevent the much more expensive failure mode where an agent confidently follows an outdated convention, repeats a known mistake, or spends several minutes debugging a problem the team already solved.
The next engineering question is not whether memory helps.
It is how to make the retrieval smaller, faster, and more selective.
What about GPT-5.3 Codex?
I also started the same model-freeze run with GPT-5.3 Codex.
That run is not a negative result for GPT.
The provider hit a credit and in-flight request limit while the benchmark was running. The harness correctly discarded the affected paired cells instead of treating missing sessions as failed model attempts.
Only 40 of the 72 paired cells were admissible, so I am not using that run to claim that GPT-5.3 Codex is better or worse.
The descriptive numbers pointed in the same direction:
| Configuration | Successful runs |
|---|---|
| Bare | 47.5% |
CLAUDE.md |
30.0% |
| RE-call | 50.0% |
But the run needs to be repeated with the provider capacity issue fixed before it becomes a valid model comparison.
That is an operational failure, not a model conclusion.
What this proves
This experiment does not show that memory makes an agent generally smarter.
It shows something narrower and more useful:
A production memory layer can improve the probability that an agent completes a real repository task correctly.
In this run, RE-call improved success by 22.2 percentage points over the static project file.
It also improved success over the bare configuration, although that gain was smaller at 8.3 points. That difference is important. The main value is not simply adding more instructions. The value is retrieving the right project-specific context when it matters.
The first experiment showed that memory can make answers more correct.
This experiment showed that the improvement survives contact with the repository.
That is the result I was actually waiting for.
What comes next
The next step is a clean competitor comparison:
- rerun GPT-5.3 Codex with sufficient provider capacity
- compare RE-call against other memory systems
- keep the same tasks, seeds, checkers, and admission gate
- publish discarded cells and operational failures alongside the scores
The goal is not to produce another flattering demo.
The goal is to find out whether a memory layer remains useful when the models, vendors, and retrieval systems change.
That is the benchmark I want to run next.
GiulioDER
/
RE-call
Memory that abstains instead of guessing: agent memory on your own Postgres with a verdict, confidence and provenance on every hit, and a calibrated refusal when nothing clears the threshold.
Memory that abstains instead of guessing.
RE-call is agent memory on your own PostgreSQL with pgvector: every hit carries a verdict
confidence and provenance, a retracted claim comes back marked superseded, and a
question the corpus cannot answer is refused rather than answered from the nearest neighbour
ATM-Bench Recall@10 92.8924 against 79.09 for the best published row (limits) · second of ten on MTRAG correct refusals (limits) · zero memory-layer LLM calls to build memory, where Mem0 pays one per session (limits)
Why RE-call · Quickstart · How it works · Product surface · Documentation · Evidence
Setup guide: install, configure and run RE-call · Validity Frontmatter: the open spec RE-call implements
Why RE-call
Nearest-match retrieval cannot tell the difference between what is true and what merely reads like it. When a corpus keeps its history, and real agent memory does, the retracted…
Top comments (10)
This is the kind of agent benchmark I trust more. The paired tasks and deterministic checker make it harder for a memory layer to win by sounding right while still breaking the repo. The zero discarded cells detail matters too, because that is where a lot of small evals accidentally sand off the ugly cases.
Thank you. Part 3 turned on the control two readers here asked for, and the headline is that the memory instruction on its own, with nothing behind it, costs 17 tasks of 358:
dev.to/gde03/i-gave-claude-code-a-...
The
CLAUDE.mdarm losing to bare by 13.9 points is the result I would chase next, and noise versus competing instructions is separable with one more arm. Run the same tasks with a length-matched placebo file: same token count, project-shaped prose, none of it applicable to the task. If the placebo also lands near 36%, the loss is context dilution and it scales with file size; if it stays near bare, the content is actively misdirecting and the length is incidental. That distinction changes the advice, because one reading says trim the file and the other says the file is wrong.I agree with this diagnosis. The 13.9 point loss is currently confounded because CLAUDE.md changes both the content and the context length.
I evaluated the benchmark implementation and added an oracle_memory arm. It injects the exact relevant corpus evidence before the agent starts, using neutral evidence formatting, with no MCP tools or filesystem memory exposed. It is a ceiling control, not a ranked product.
I also added recall_prefetch, where the harness queries RE-call with the exact task prompt and injects the returned evidence. This separates retrieval quality from the agent’s decision to initiate a memory search.
The next comparison should therefore be:
bare → CLAUDE.md → length matched placebo → oracle_memory → recall_prefetch → natural recall
The interpretation becomes much cleaner:
I ran a different control first, and it changed the question: the memory instruction with no memory behind it loses 17 cells of 358 against bare, p = 0.053.
Your length matched placebo is in the eight arm run that started today, so dilution versus misdirection gets a real answer rather than my hypothesis.
Part 3 is the instruction only result:
dev.to/gde03/i-gave-claude-code-a-...
Love this experiment! Using strict test suites as the ground truth feedback loop makes the whole evaluation so much more practical. Super interesting results.
Thank you. The checker earned its keep in part 3: the previous run's headline finding turned out to be my own harness handing one arm a bigger prompt, and an executable test meant I had to withdraw it rather than argue about it.
dev.to/gde03/i-gave-claude-code-a-...
Disclosure first: I build cachly, a memory layer for coding agents in the same space as RE-call - so read my enthusiasm with that bias in mind.
Three things here deserve more attention than they'll probably get:
The CLAUDE.md arm losing to bare by 13.9 points is the most important number in the post. It matches what we see from the retrieval side: the value is not "more instructions," it's the right project fact arriving only when the task needs it. A static file pays its token cost and its distraction cost on every task; retrieval pays only when consulted. One suggestion for your 83.3% / 85.0% / 70.8% funnel: split the remaining failures by class - didn't-search / searched-found-nothing / found-the-wrong-thing / found-the-right-thing-and-ignored-it. When we ran that split on retrieval failures, one class dominated everything else, and the fix is different for each class - so the split decides what you build next, not just how well you did.
A question from a scar of ours: were the checkers themselves tested against known-bad repositories? We've been burned by deterministic guards that stayed green when their input silently died - a checker that cannot go red on a wrong repo is decoration, and it fails in the direction nobody audits. With 24 tasks it's an afternoon: one deliberately wrong repo state per task, checker must fail. Zero discarded cells is a great admission gate for the agent side; known-bad twins are the same gate for the judge side.
The six-arm design in your reply to Vinh is worth its own article - and I'd add a seventh: wrong_memory. Inject plausible but outdated evidence (the stale rate-limit memo from your own demo). Every other arm measures how much good memory helps; that arm measures how much bad memory costs - which is the number your abstention thesis actually lives on. If refusal is calibrated, wrong_memory should degrade toward bare, not below it.
On superseded-marking: we landed in the same place independently last week - a correction must name what it replaces, both sides get marked, and a dangling reference is refused at write time. Which makes me curious how RE-call handles the dangling case: what happens on retrieval when the superseded target was later deleted?
The failure split ran and your prediction held: one class dominates. On superseded, 45 percent of sessions retrieve neither planted document, which is the largest single loss in the benchmark.
Checkers: yes, every task carries a known bad twin in CI. 222 parametrized cases, and only the informed reference is allowed to pass.
On the dangling case, it resolves to
ambiguouswithunresolved_supersession_referencerather than promoting the survivor. Going to check that for you, I found the diagnostic appears once in the repository and in no test. You found a hole.Part 3:
dev.to/gde03/i-gave-claude-code-a-...
The dangling-case diagnostic appearing once and in no test is the "guard that has never fired" class - the resolver does the right thing and nothing would notice if it stopped; a twin test that plants exactly that dangle would close it, and your 222-case known-bad harness is clearly the place it belongs.
The 45 percent holding as the dominant class settles where the work is, so: we would like to be measured. I build cachly (disclosed upthread); adapter issue incoming this week. Two commitments up front, so the run means something: we will clone the harness and run privately first - the public run should not be our rehearsal - and one pre-registration against ourselves: we expect your superseded condition to hurt us, because our store currently lets successes overwrite while corrections merely append, a defect we documented against ourselves last week. If your harness confirms it, you will have measured our backlog, and that is worth more to us than a flattering row.