ChatGPT, Claude and Gemini all ship memory features now, and the pitch is the same: an assistant that knows you. Your preferences, your projects, your goals, carried across conversations.
OpenAI's own memory FAQ admits the awkward part. It says the previous saved-memories system "often became stale" and could hold contradicting memories at the same time. That is exactly the version of the problem a new benchmark called Memora is built to measure.
The paper behind it, "From Recall to Forgetting" (Uddin et al., Arizona State, University of Arizona, Genies, Findings of ACL 2026), argues that the way we test agent memory has a blind spot: we test recall, while in the authors' framing, real long-term memory is mostly revision.
The blind spot in how memory gets tested
Look at how existing long-term memory benchmarks are built:
- In LoCoMo, 94% of evaluation questions can be answered with evidence from at most two past sessions. In LongMemEval it is 85%. Across the major benchmarks, the average question needs about one session of history.
- Updates barely exist. LongMemEval allows at most two knowledge updates before evaluation, and PersonaMem handles changes across no more than three sessions.
So most memory scores measure something narrow: can the system dig up a fact that was stated once and never changed? Real use does not look like that. People correct themselves, change their minds, finish projects, cancel plans. Information gets updated, withdrawn and replaced, over weeks and months. A memory system that never forgets is not automatically impressive. It can be a liability with good recall.
What Memora does differently
Memora simulates 10 professional personas (engineers, researchers, designers, executives) across three timeline settings the authors call weekly, monthly and quarterly. Those labels refer to the span of the simulated history: about 155, 615 and 1,991 sessions per persona respectively, at roughly 16 turns per session.
The important part is what happens inside those sessions. In the quarterly setting, a persona's memory sees about 1,171 memory operations on average: 63% additions, 18% updates, 19% deletions. Preferences drift, activities repeat, goals progress. Every change is tracked in an explicit memory trace, so the benchmark knows exactly which facts are current and which are stale at question time.
Three task types get evaluated:
- Remembering: recall previously discussed facts.
- Reasoning: combine several memories to derive an answer (like computing budget status from logged expenses).
- Recommending: suggest things that match the user's current preferences, not their old ones.
Scoring uses a metric the authors call FAMA (Forgetting-Aware Memory Accuracy). Standard accuracy asks: did the response include the right information? FAMA also asks: did it avoid the outdated information? Each question comes with "must include" criteria and "must exclude" criteria, and using an invalidated memory costs you. Three LLM judges vote on each criterion, with 88.3% average agreement against human annotators.
They tested two setups. Four LLMs (GPT-5.2, Claude Sonnet 4.5, Gemini 3 Pro Preview, Qwen3-32B) reading conversation history directly, with and without reasoning tokens. And six dedicated memory agents (A-Mem, LangMem, Mem0, MemoBase, MemoryOS, Nemori) that store and retrieve memories across sessions, all answering through the same GPT-4o-mini backend. That last detail matters for how you read the results: the agents run on a smaller model than the LLMs reading raw history, so this is memory machinery on a fixed, modest backend, not a controlled memory-versus-no-memory comparison at equal model strength.
What they found
One note on the numbers below: the authors sum per-question FAMA across all three timeline settings, so the aggregated scores run to 300, not 100. Read them as relative scores, not percentages.
In this benchmark, memory systems are clearly better at remembering. On aggregated FAMA, memory agents nearly double the LLMs on recall tasks: 119.5 versus 65.7. Explicit memory stores do their core job.
LLMs match them on recommending anyway. 144.7 to 153.3 for plain LLMs against 138.4 for memory agents. The authors' explanation: recommendations tolerate approximate memory, because you can guess plausible preferences from persona cues without retrieving anything precise. So in this study, the task where memory should matter most is the one where it added the least.
Reasoning over memory is weak everywhere. Best average is 27.6, from the memory agents. LLMs sit at 12.4 to 13.9. In the authors' manual error analysis, every sampled reasoning error traced back to partial retrieval: the system had some of the pieces and answered anyway. One example from the paper: after logging several expenses against a monthly budget, agents said "within budget" instead of computing the remainder. Reasoning tokens made little difference.
The forgetting penalty grows as memory scales. This is the finding worth sitting with. For LLMs reading raw history, the FAMA penalty from stale-memory use shrinks on longer timelines (32.6 points in the weekly setting, 17.8 in the quarterly one), but for an unflattering reason: on long timelines the old conversations fall out of the context window entirely, so the model physically cannot reuse them. Memory agents show the opposite trend. Their penalty grows from 18.2 to 29.5. The more history the memory store accumulates, the more the agent leans on facts that stopped being true.
This changes rankings. In the monthly setting, MemoryOS (112.8) and A-Mem (112.0) both beat Nemori (105.4) on plain memory accuracy. But Nemori's forgetting penalty is much smaller (15.4 versus 28.4 and 29.5), so once FAMA counts stale memory, Nemori jumps ahead of both. In this result, the "worse" memory system gives better answers because it is better at letting go.
The error analysis makes it concrete. Of the recommendation errors the authors sampled, 64% came from outdated memory that was never invalidated. The paper's example: a user preferred non-fiction, later shifted to contemporary fiction, and the agent recommended a historical biography. It remembered. It just remembered the wrong version of the person.
What this means if you are building with memory
None of this is a reason to skip memory. It is a map of where the hard part actually is.
- Treat memory writes as versions, not appends. An update should supersede the old fact, and retrieval should prefer the current state. A flat, ever-growing list of "facts about the user" is exactly the shape this benchmark punishes.
- Test with corrections, not just recalls. If your eval set has no updates or deletions, you are measuring the easy half. Add "the user changed their mind" cases and check what the agent does with both versions.
- Score the exclusions. FAMA's core trick is cheap to copy: for every eval question, write down what the answer must include and what it must not bring up. Stale-memory use is invisible until you penalize it.
- Do not expect a memory layer to fix reasoning. Retrieval and consolidation came apart in every system tested, and the gap between them is the weakest point.
The honest caveats
The conversations are simulated by LLMs, not collected from real users. The authors argue this makes the benchmark a lower bound: real interaction adds ambiguity and contradiction, so a system that fails here will likely fail harder in the wild. Fair, but still synthetic.
As noted above, all six memory agents answered through GPT-4o-mini, so the agent numbers reflect the memory machinery more than the model. The scoring runs through LLM judges, though with three-way majority voting and a human agreement study behind it. Ten personas and three memory types (preferences, activities, goals) is also a narrow slice of what long-term memory could mean.
Even with all that, the direction of the result is hard to wave away: everyone is racing to give agents more memory, and Memora's results suggest that more memory, badly maintained, makes answers worse.
Sources: Uddin, Shubham, Blanco, Baral, Wang, "From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents", Findings of ACL 2026. Paper: https://aclanthology.org/2026.findings-acl.1337/ (preprint: https://arxiv.org/abs/2604.20006). Code and data: https://github.com/geniesinc/Memora. Product memory docs: https://help.openai.com/en/articles/8590148-memory-faq , https://support.claude.com/en/articles/11817273-use-claude-s-chat-search-and-memory-to-build-on-previous-context , https://support.google.com/gemini/answer/16598469
Written with AI assistance.
Top comments (21)
The Memora split is the right one — 18% updates and 19% deletions in the quarterly setting means revision is a third of memory work, yet benchmarks barely test it. I'd push one step further: deletion shouldn't be silent either. A memory that quietly discards a 'cancelled plan' without surfacing the change can hallucinate intent just as badly as one that remembers too much. Does the benchmark score whether the system announces what it forgot, or just whether the fact is gone?
It scores the answer, not whether the system announces the deletion. Sections 3.5 and 4 define criteria for what a response must include and which outdated facts it must exclude. So a good score doesn't prove the stored fact was erased, either. Your cancelled-plan example adds a separate test: does the assistant acknowledge the change and make clear that the plan is no longer active? I'd keep that distinct from simply not reusing it in a later answer.
That distinction is exactly right — scoring the answer and proving the erasure are two different claims, and a clean response can hide a stale store. Keeping the acknowledgement test separate from the non-reuse test feels like the honest split: one checks what the user is told, the other what the system still holds. Do you think the acknowledgement belongs in the benchmark itself, or as a separate transparency check alongside it?
I'd put it alongside the benchmark, with a separate score rather than fold it into FAMA. That would keep three checks visible: did the assistant acknowledge the change, did later answers stop using the old fact, and was the stored memory actually removed or invalidated? One correction to the split: non-reuse tests behavior, not what the store still holds. Checking the store needs its own inspection, and a reassuring acknowledgement shouldn't count as proof of either.
Fair correction — non-reuse is behavior, and behavior can look clean while the store still holds the old fact, so it can't stand in for inspection. Your three-check split is the honest shape: acknowledgement, non-reuse, and store invalidation each scored on their own, so a good score on one can't launder a failure on another. The store check is the one I'd expect vendors to resist most — how would you inspect it without their cooperation, an export, or a probe that forces the old fact back out?
Without access to the store, I wouldn't claim to have verified erasure. I'd mark that check as unverified and report the behavioral results separately. Probes can expose reuse, but getting the old fact back doesn't tell you whether it came from memory, retained chat history, or inference; not getting it back doesn't prove deletion. A store-level claim needs inspectable evidence with a clear scope, including what an export or audit leaves out.
That's the honest label — 'unverified' beats a probe result dressed up as proof, especially since a probe can't tell memory from chat history or inference. The scope point is the one I'd carry forward: an export or audit is only as good as its list of what it doesn't cover, and that list is usually where the stale fact survives. Would a scoped claim — deleted from these stores, with these caches explicitly out of scope — be enough for the benchmark to accept, or does it need to stay unverified until the scope is complete?
For the separate store check we're discussing, I'd accept a scoped result only if the scope was fixed before testing and the evidence supports it. "Removed from these stores; these caches not checked" is useful, but it isn't a system-wide deletion pass. Any excluded cache that can still feed an answer leaves that broader claim unverified. Memora itself doesn't define this acceptance rule; this would be an additional evaluation, not a reading of FAMA.
Pre-registered scope is the key condition — a scope fixed after testing is just a post-hoc excuse list. And your point about any excluded cache that can still feed an answer is the real test: the scope statement has to name not just what's checked, but what could still leak. Would you require that leak-path list as part of the claim, or is naming the unchecked caches enough?
I'd require the known paths back into an answer, plus an explicit note about unknown ones. Naming an unchecked cache doesn't say whether retrieval can still read it or whether it can repopulate a checked store. For a narrow claim, the report should show that boundary and its evidence. It still shouldn't imply a complete inventory of every possible path, or turn a partial check into a system-wide pass.
That boundary framing is the upgrade — a scope list alone doesn't say whether an unchecked cache can still feed retrieval or quietly repopulate a checked store, so the report needs the path, not just the name. The explicit 'unknown paths' note is what keeps a narrow claim honest: it stops a partial check from reading like a system-wide pass. Would you want evidence per path — a retrieval trace showing the cache is actually cut off — or is a declared, pre-registered boundary enough for the benchmark to accept?
Evidence, not just a declaration. Pre-registering the boundary says what will be tested; it doesn't show the boundary held. A retrieval trace can support that claim for the requests observed, but one clean trace isn't proof that a cache is unreachable. I'd keep the conclusion at the level the evidence supports: tested paths passed under stated conditions, with untested paths still unverified. That's a separate audit proposal, not something Memora currently certifies.
That's the right ceiling — a trace shows the boundary held for the requests observed, not that the cache is unreachable in general, so the honest conclusion is exactly 'tested paths passed under stated conditions.' The distinction that matters is evidence of absence versus absence of evidence: one clean trace is the second, and dressing it up as the first is how a partial check becomes a system-wide pass. If this became a separate audit proposal, would the trace need to be continuous — sampled across real traffic — or is a bounded, pre-registered probe set enough to accept?
Stale facts are a cost too. They ride along in every prompt until someone prunes them, and the spend grows even when the answers look fine. A retrieval that returns only what is still true cuts both the errors and the tokens.
Yes, when stale facts get retrieved into the prompt, they take up space even if the answer happens to be right. I'd measure that separately from accuracy: how many invalidated facts reach the prompt, and how many input tokens they add. Memora doesn't report cost or latency, so it doesn't establish the net savings once the work of checking and updating memory is included.
The "must exclude" criteria in FAMA are what I'd want in every memory eval. Recalling a fact is easy to test; noticing it was withdrawn three months ago is what actually goes wrong in practice. "A liability with good recall" is a good way to put it.
Thanks. That's the useful shift: an answer can contain the right current fact and still get something wrong by adding the withdrawn one. Checking both what belongs and what no longer belongs makes that failure visible.
The Nemori result in the monthly setting is the one I would check. MemoryOS (112.8) and A-Mem (112.0) beat Nemori (105.4) on plain accuracy, but Nemori's forgetting penalty is 15.4 against 28.4 and 29.5, so the order flips under FAMA. That flip rests on about 13 points of penalty difference against 7 of accuracy. With three LLM judges at 88.3% agreement with human annotators, roughly one criterion in eight is graded differently from a human. Does the paper report agreement for the "must exclude" criteria separately from the "must include" ones, and any interval on the per-system scores? Without those, I would read the Nemori ordering as one run of one judge panel.
Appendix C.2 reports agreement by judge-consensus pattern, not separately for presence versus exclusion criteria. The 88.3% comes from 100 criteria sampled equally across four consensus patterns, so it isn't a benchmark-wide error rate or an uncertainty interval on the ranking. I couldn't find confidence intervals for the per-system scores. Figures 3 and 4 have error bars, but those show variability across timelines or tasks, not uncertainty on that monthly ordering. I agree the flip should be read as a reported result under this evaluation setup, not a statistically established ranking. That caveat belongs beside the comparison.
The Nemori flip is the finding that stuck with me. A system that forgets cleanly beat two stronger recall systems once stale answers counted against them.
Exactly. In that monthly comparison, the ranking changes once outdated information carries a penalty. I'd call it better avoidance of stale facts in the answers, though, rather than proof that the underlying store forgot cleanly. That's what makes the result useful without asking it to prove more than it measures.