I work at Wontopos, so I'm biased. I build a memory API. But this post isn't about it. It's about a problem I keep reading about, and I want to hear how you deal with it.
The problem
An AI that remembers things will sooner or later remember something that is no longer true.
Here is a made-up example, just to show the shape of it:
| When | What the user says | What gets stored |
|---|---|---|
| March | "I live in Seoul." | lives in Seoul |
| July | "I moved to Busan last month." | lives in Busan |
| October | "Find me a good dentist nearby." | ??? |
Both memories are true statements. One of them stopped being true. If the memory layer only looks for text that is similar to the question, both come back, and the model has to guess which one wins. Sometimes it guesses wrong, and it sounds confident when it does.
That's the part that bothers me. A missing memory is easy to notice. An old memory that sounds right is not.
Why it's harder than it looks
- Similar is not the same as current. Search finds what matches the question. It doesn't know which fact replaced which.
- Adding more memory makes it worse. Every correction is one more thing that can contradict an older entry.
- Nobody tells the system. People don't say "forget what I said in March". They just mention the new thing.
- The failure is quiet. Nothing crashes. The answer is just a little wrong.
What people do about it
From what I've read, people seem to do one of these:
- Edit the notes file by hand.
- Tell the AI "that's wrong, now it's X" and hope it overwrites.
- Wipe everything now and then and start again.
- Store everything with timestamps and let the model sort it out.
What we do
In our API, an outdated memory is replaced on purpose with a supersede call, so the old value is marked as replaced and not just buried. We also keep revisions, so you can see what changed. I don't want to oversell this. Deciding when to call it is still your job, and I'm still thinking about how to do that part better.
We also put a STALE axis in Glasshouse, an open memory benchmark I'm working on (still a draft), because I couldn't find an easy way to measure this. Details are in the repo if you want to look.
My question
I'd like to hear from people who build or run their own memory:
- Has your agent ever answered with something that used to be true? What was it, and how did you notice?
- How do you decide a memory is outdated? Timestamps, a rule, a human, the model itself?
- Do you overwrite the old value, or keep both and mark one as old?
You don't need to sign up for anything. The details are at wontopos, and the site has a short survey on the main page. A comment here is just as good, and the survey is a bonus.
Top comments (2)
The Seoul-to-Busan example nails why this failure is quiet — both memories are true statements, so similarity search returns a confident wrong answer instead of an obvious miss. Marking the old value superseded rather than burying it is the right call, since revision history is what lets you audit why the dentist recommendation changed. The hard part you flag honestly is deciding when to supersede: do you trigger it on explicit contradiction only, or does the STALE benchmark also catch the slow drift where nobody ever states the new fact outright?
In my own pipeline every stored answer carries the date it was written. When the offer changed, I stopped reusing the answers from before that date instead of editing them.