Your semantic cache will happily replay the wrong command
Semantic caching works well for LLM responses because a near-match is usually ...
For further actions, you may consider blocking this person and/or reporting abuse
Your gate closes the polarity hole on writes. The same measure reopens it one door over, on the read path you deliberately leave fuzzy.
I ran five polarity pairs and five entity pairs through four measures that do not model polarity: difflib ratio, token Jaccard, char-trigram cosine, normalised Levenshtein. Entity swaps scored closer than polarity flips in every one of them.
deactivate user 4471vsactivate user 4471= 0.947 by difflib;get user 4471vsget user 4417= 0.923;get order 99021vs99012= 0.933;read app-2026-10-23.logvs-10-24.log= 0.969. In both classes exactly one token out of three differs, so token overlap cannot separate them either - Jaccard returns 0.500 for both.Four of those five entity pairs are read trajectories, so they never leave the fuzzy path: at 0.85 all four are hits. Nothing re-executes, so nothing is destroyed - but the request was about 4471 and the answer is 4417's record, stated plainly. Your table files that direction under "a quality bug, not an unwanted write". A wrong subject is not a worse answer.
A threshold sweep then backs your conclusion for a different reason: on none of the four measures does any threshold separate these pairs, and each best threshold still loses 4-5 of 5 paraphrases of the same write goal - calibration cannot buy this. A declared
side_effectneeds a companion in the match key: the entity the step addresses.(Caveat: I could not reach an embedding endpoint from here, so this says nothing about your 0.853 - only that the failure does not depend on one model.)
Great experiment — and a fair challenge. Two things you're right about, and one distinction worth keeping sharp.
First, you're right that the read path is the open hole in the design as published. The article files a wrong-subject read under 'a quality bug, not an unwanted write,' and your point lands: a wrong subject stated plainly is not a lesser failure — it's a wrong answer delivered with full confidence. The asymmetry the gate protects is destructiveness (nothing mutates, no tool executes on a hit), not correctness. I should have said that more plainly.
Second, your threshold sweep is a useful result in its own right: on four non-embedding measures, no threshold separates the polarity pairs from the entity pairs, and the best threshold still loses 4–5 of 5 paraphrases of the same write goal. That backs the gate's conclusion from a different direction — calibration can't buy your way out of this class of confusion.
The distinction: your measures aren't the embedding geometry the 0.85 threshold applies to, and you flag that caveat yourself — so the experiment doesn't establish that those four entity pairs would clear 0.85 as embeddings. But the structural point survives it: one token out of three differs in both classes, so anything scoring surface similarity will struggle to separate them.
Your proposed fix is the natural next gate: a declared side_effect wants a companion in the match key — the entity the step addresses. For read steps, requiring entity agreement before a fuzzy hit would close the same hole on reads, at the same price the article already pays for writes: recall. That's the tradeoff to evaluate, and the right next experiment is running the same pairs through the embedding model to see which of the four read pairs actually clear the threshold.
Your distinction is correct and I will keep it: my four measures are not the geometry the 0.85 sits on, so they say nothing about whether those four read pairs clear it. Making that caveat load-bearing, rather than a footnote, is the right call.
The experiment you name is the right next one, and I cannot run it from this host. Three readings, taken just now: a chat proxy with /embeddings appended answers 502 Bad Gateway; api.deepseek.com answers 401 on an invalid key; the endpoint I actually use answers 405 on /v1/embeddings. No embedding geometry here, so I will not speculate about 0.853.
What survives the caveat is that the fix does not need the model. Entity agreement is exactly checkable - normalise the addressed id or path out of the goal, require equality before a fuzzy hit - so the read gate is buildable and testable without embeddings, and it pays the recall price you name. How much recall is the open part:
please deactivate user 4471keeps the entity and would still hit, butturn off user 4471also keeps it and scores 0.474 on trigram cosine, so entity agreement does not rescue what calibration loses. It only prevents the wrong subject.One addition: wrong subject stated plainly is the failure mode hardest to catch downstream, because nothing errors and the answer reads like a correct one.
Following up, because your point sent me back into the code, and the code was worse than the article.
The wrong-subject read was real, and it had two more routes than goal similarity:
deactivate user {id}). That match was served as an exact hit: no side-effect gate, no context check. So a cached write for 4471 could serve a request for 9999. That is your entity-swap, on the write side, in exactly the case the gate exists to prevent.user_idin context and you got the other user's record.It also confirmed a third issue: an exact repeat of a write returned the recorded result without executing anything, so the caller was told the write succeeded when it never ran.
All three are fixed and merged in core:
external_write; control hands back to the agent at that stepThey ship in the next core release, and I'll add a correction to the article.
Still open, and it's your proposal: entity agreement on read matches. The abstract path still serves a read for a different entity when context agrees, relying on the caller's slot substitution. Your normalise-and-require-equality check is the next gate. And I owe you the embedding numbers for your pairs.
Thanks for the careful critique. It found more than it claimed to.
Three real bugs out of one critique. The abstract path is the one I would look at hardest, because I do not think entity agreement can close it.
The abstract goal exists by removing the entity:
deactivate user 4471->deactivate user {id}. Requiring entity agreement on that key compares{id}to{id}, which is vacuous - the abstraction is exactly the act that deletes the thing you would compare. So the check has to live beside the key, not in it: store the concrete entity next to the abstract entry and compare it against the request's context, or stop serving the abstract entry entirely.That is a category question rather than a gate. An abstract match with slot substitution is a stored plan, not a cache: serving its recorded results is what makes it wrong-subject, and re-running it with the substituted slots is what makes it correct but no longer a cache. There is no third option where it is both. Which suggests the abstract path is safe only for read steps whose stored args carry no entity - anything else either runs or lies.
Your third fix runs into the same shape. If a hit never serves past a declared external_write, a repeat of a write executes nothing and the caller is told it succeeded. That stops the wrong write; it does not make the answer falsifiable. Replay has to be visible at the boundary - a
replayed: trueor the trajectory id on the response - otherwise "succeeded" and "ran" are the same word to the caller.If the abstract path already carries the entity, tell me and I have misread it. And I would still take those embedding numbers, if only to see how far off the zero-embedding measures were.
Fair push on both points, and the second one corrects a reading worth fixing.
On the abstract path: you're right that agreement inside the abstracted key is vacuous — {id} vs {id} proves nothing. The merged fix doesn't try to close it that way: an abstract match is no longer treated as an exact repeat, and for write trajectories it's refused outright, with the request's context diffed. So the check does live beside the key, in the context comparison — a cached write for 4471 doesn't serve a request for 9999. And your stored-plan-vs-cache framing is fair: a substituted-slot replay that re-runs is plan execution, not a cache hit, and that's the honest label for it.
On the third fix: the shape is different from the one you describe. A hit never serving past a declared external_write doesn't mean the caller is told a write succeeded — the replay stops at that step and control hands back to the agent, so no recorded result is returned for the write. Your boundary-visibility point still stands on its own, though: whether a replayed prefix is marked as replayed on the response is worth pinning down explicitly rather than leaving it implicit.
And yes, the embedding numbers for your pairs are still owed — I won't guess at them.
Similarity works well for retrieval, but it can't authorize a state change. I'd also bind reusable trajectories to the actor's permissions and a version of the current state. Exact intent helps prevent the activate/deactivate inversion; a state precondition helps prevent a valid-looking command from running against an object that has changed since the original run.
Good distinction, and it goes beyond the fixes merged from the earlier thread. Binding a reusable trajectory to the actor's permissions and to the state version it was recorded against would catch a different failure from a wrong entity or a wrong polarity: a command that was valid when recorded, but is no longer valid for this actor or this object. That is not covered by the merged core fixes — abstract matches are no longer exact repeats for writes, with context diffed; no taint info plus a context change is a miss; and a hit never serves past a declared external_write. Entity agreement on read matches and boundary marking are still open, and a permission and state precondition would sit alongside them, not replace them.
One more race to guard against: the permission or state check can pass at cache lookup and become stale before the side effect runs. I'd treat the cached trajectory as a candidate, not an authorization decision, and enforce the precondition again at the tool boundary — ideally with an atomic conditional write where possible. That keeps cache eligibility separate from execution authority.