I keep hammering the point that any coding-agent score is model + harness, not model alone. Same context-carryover rules, same note convention, same tool loop, same judge, or the comparison is garbage.
Engrim (github.com/timgordontg/engrim) is a useful reminder that there's a third axis I've been underselling: state.
It's a local-first SQLite memory engine for Claude Code, Cursor, Windsurf, Codex. Project-scoped, embeds records plus SQLite FTS, persists decisions and rationale between sessions. Hybrid retrieval, no cloud. Cross-session recall is the whole pitch.
And here's the thing: that pitch only works if the agent actually gets better as its memory grows. Engrim exists because the authors believe persistent project context beats a cold context window on every new session.
Which is exactly what almost no benchmark measures.
Look at how agent evals are actually built. They load a repo, drop you in at an issue, and score the patch. Empty CLAUDE.md. No cursor rules. No memory file. Every run starts the model at the same amnesia. That's deliberate, for reproducibility. You can't run a score if the model's head is full of last run's secrets.
But clean reproducibility bought a distorted measurement. The agent you benchmark cold is not the agent your team runs after a month of accumulated project memory. The more an agent keeps between sessions, the wider that gap gets. A 30-minute cold-start eval tells you almost nothing about how an agent with six months of ingested project decisions will handle a real migration.
This is the same structural blindness as correlated judges scoring a session: you've measured one blind spot and called it a committee. Here you've measured one memory state (empty) and called it the model. The variable that actually drives production output, prior state, is exactly the variable the eval pins to zero.
The fix doesn't have to destroy reproducibility. You can parameterize it instead: run a task three ways, empty context, a small hand-curated project memory, and a large one. Watch the score move. Tell me which tool's score moved the most, and I'll tell you which one you should run in a long-lived repo. That result is more useful than a hundred clean cold-start tables.
Engrim's not the product here. It's the tell that this axis matters and nobody's benchmarking it. If your eval measures amnesiac models, its scores are a floor, and the gap to production is your real signal.
Top comments (3)
The other side of running persistent project memory across fifty sessions is cache invalidation. When a cold agent starts empty, it fails by missing context and is forced to grep the live tree. When a warm agent loads an SQLite store containing a dead schema from three weeks ago, it hallucinates with high confidence because the retrieved record looks authoritative.
Testing against empty, small, and large memory tiers is a good start. The eval suite also needs a poisoned or stale memory variant to measure whether the harness can detect when local disk state contradicts the actual codebase.
A useful fourth tier is contradictory memory with explicit provenance: one stale design decision, one recent code observation, and a task that forces a choice. The agent should cite the conflict, inspect the live source, and update or retire the record. That measures whether persistence becomes a maintained system of evidence instead of a stronger source of confident drift.
The three tiers measure the wrong difference for the ranking you want to do at the end. A hand-curated project memory is written by someone who already knows the repo, so on some fraction of tasks it contains the decision the task is asking for, and the delta from empty then mixes "the agent retrieved and used context" with "the answer was sitting in the store". Ranking on which tool's score moved most rewards whichever tool retrieves most aggressively on exactly those tasks, and none of the three arms tells the two apart.
The control that does is a fourth arm built as a null rather than a stress test: a decoy store of the same size and format, curated the same way, but drawn from a different project. Real recall reads as a gain on the matched store and flat on the decoy; a gain on both is a context-bulk or formatting effect that will not survive contact with a repo the store was never built for. Rank on matched-minus-decoy, not matched-minus-empty.
That is a different question from the stale and contradictory tiers already suggested in this thread. Those ask whether the agent notices its store is wrong. The decoy arm asks whether the improvement you are quoting came from retrieval at all, and it has to be answered first, because a tool that scores well on decoy memory has no recall result to be robust about.