Yesterday afternoon a runtime I wrote created 70 paragraphs on the same draft article. It was supposed to create one. The runtime was an experiment in explicit state, the kind where the model gets the task spec, a small JSON, the last observation, and nothing else; it had a commit rule I was proud of: nothing counts as done until a read-back confirms it. After every paragraph it read the draft back, found no paragraph, and asked the model for another one. Seventy times.
My first suspect was the model. It was the last thing that had produced output, and blaming it costs nothing. Then I looked at the response sizes of the read-backs: 742 bytes, 742, 742, twenty-five times in a row, then 131,991 bytes at once. The paragraph list had been served from a cache with a 60-second lifetime. The model had done exactly what a correct runtime told it to do, and the runtime had been lied to by its own read path. Not the model. Not even the rule. The order of two reads.
One line of context so you know where I stand: I run engineering at GoodBarber, an app platform, and I operate a production MCP server plus a few scheduled agents of my own. In the write-safety piece I asked whether a 200 meant the right thing happened; in the determinism piece I argued that verification is the job now. This one is about what comes after the verification says no. A system that is allowed to be wrong cannot be debugged by asking whether it was wrong. It has to be debugged by three questions, in this order: what was it allowed to get wrong, which layer got it wrong, and how long did the error live.
Write down what it is allowed to get wrong
I had never written this down. Doing it took an evening and changed what I look at first.
| block | output | tolerated | never allowed | detected today by | when it does not run |
|---|---|---|---|---|---|
| MCP server, as its operator | tool results with a policy envelope, one usage row per call | latency; a read served from the 60-second cache when no write preceded it; a client choosing the wrong tool; 429 above 1,000 calls a minute | a 2xx for a write that did not persist; a stale read right after a write by the same token; a tool silently missing from a client's list; a push nobody scheduled | Sentry on unhandled exceptions; status and error code in the usage row; one guard test against schema shapes that hide tools | loud: 421, 503, 429. Silent: the server is up and its cache is wrong |
| content agents, Claude Code sessions writing to seven blogs | drafts in nonpret, patches, JSON-LD |
a clumsy sentence; a meta title to redo | a live article patched without a diff; a create that defaults to published; an em-dash in a slug; invalid JSON-LD on a live page; an empty body reported as done |
my approval in the CMS; a diff before and after when a patch touches live pages; JSON-LD validation after injection | nothing ships. The risk is the half-written object |
| scheduled tasks: noon ops review, two code reviews, glossary sentinel | a DM to one colleague; two morning reports; a Monday report on 170 pages | a false positive dismissed in a minute; a missed run if the next one says so | "all green" over a broken page; a number in the DM that is not in Nagios; a skipped run with no trace | 22 negative test cases and a positive control on the sentinel; the colleague reading the DM; nothing counts the runs | silence, indistinguishable from "nothing to report" |
Two admissions fell out of the table. First, I have no fixed review routine: I read what an agent produced when it produced it, or when something surfaces, with no schedule and no written verdicts. Second, half of the "never allowed" cells had no detector the day I wrote them. Write it down, or every output is a bug and a feature at the same time.
Attribute before you debug
Here are the incidents I could date on my own systems since April, coded by the layer that was actually wrong. The list is what I could date, not a sample, and the split describes this list, not my fleet.
| when | what | layer | first suspect |
|---|---|---|---|
| April 10 to 14 | access tokens set to 300 seconds "just in case"; every client logged out every five minutes | configuration | the clients |
| April 8 to August 5 | a per-app session cap counted sessions for an hour after their last call; clients that never sent the protocol DELETE starved it and legitimate traffic got 429s | harness | the clients, again |
| June | agents patched translations on seven live blogs with no draft step; a diff afterwards found 46 damaged paragraphs (from my notes; the artefacts are gone) | harness | the model |
| June | a translation pipeline reported green on a French draft with a title and no body; I remember the run as green, its history is purged, the draft is still empty | harness | the model |
| June 5 to 22 | a schema change made for a directory review put a oneOf at the root of one tool; clients that drop such tools stopped seeing it for 17 days |
tool contract | the clients |
| July 29 and 31 | an unbounded Redis pool hit 10,000 clients, 3,650 connections rejected, every worker locked out; two days later one stream read per session filled the 200-slot pool | infrastructure | the load |
| August 5 | a delete succeeded and the immediate read-back returned the object; fixed on August 10 by keying the read-after-write bypass on the token instead of a per-session variable | cache | the model, for a minute |
| August | the first version of my llms.txt annex read 8 days of logs and reported them as 121 | the debugger | the logs |
| September 3 | a state runtime created 70 paragraphs because its second read went through the cache | cache, then the runtime's read order | the model |
| June 3 to September 2 | 125 calls to 34 tools that do not exist, including GBContent.getItems(sectionId, opts, onOk, onErr), from 5 apps |
the model, with a harness that let it retry | nobody, until I counted |
Ten lines. The model is the direct author of one of them, the invented tool names, and even there a client let it retry the same name until it gave up. My first suspect was the model in four of the ten. The reflex was wrong four times; the model, once.
The rule that fell out of it: attribute before you touch the prompt. Data and credentials first (was the input what you think it was), then tool and cache (did the read-back read the world or a copy of it), then harness and scheduler (did the loop do what the loop does, retry, truncate, reorder), then the model, then the human reading the output. Every layer above the model is deterministic and checkable in minutes. The prompt is where you go last, because it is the only layer where a fix cannot be verified.
The transcript is a witness, not evidence
When an agent has been wrong, the transcript is the first thing you open and the last thing you should trust. It records what the model saw and what it said. It does not record what happened. Ask the transcript "what did you call, with which arguments", never "did it work".
On September 3, 2026, I measured what our own read path answers, 50 cycles on a test app, every article created as a draft and deleted after. Create, read, delete, read: the immediate read after the delete still returned the object 3 times out of 50, after the August fix. Once in 50, the immediate read after a create did not find the new article at all. Then the sequence an agent actually produces, create, read, read, delete, list, read: the deleted object came back 50 times out of 50, and stayed for a median 61.1 seconds, maximum 61.2. The server bypasses the cache for exactly one read, the one that follows the write; a list in between resets it. A transcript of that agent would show a delete with a 200 and a read that returned the article. Both true. Neither what happened.
The same cycles confirmed three things I had logged in August as tickets. A delete returns only the policy envelope, no deleted, no id, no status; the transcript literally cannot show what was deleted. cms_create_article still defaults to published, so a create without an explicit status is a publication. And the verification hints on a create list cms_get_article, cms_get_article_paragraph and cms_get_event, which are the first three read tools of the family in alphabetical order, while the tool plan points its verification call at cms_list_cms_sections. Follow the plan to the letter and you verify an article by listing sections.
None of this is exotic, and that is the point. The June agents that damaged 46 paragraphs were not lying. They were reporting. Nobody had asked the state, only the transcript.
Measure how long the error lived
Error rates are the number everyone asks for and the least useful one I have. What changes behaviour is time to detection, per incident, per channel.
| error | lived | caught by |
|---|---|---|
| stale read after delete, list in between | 61 seconds, every time | a script, September 3 |
| the annex that read 8 days as 121 | hours | me, because a number looked too round |
| the 17-day invisible tool | 17 days | a guard test written on June 22; who noticed first is not in the log |
| 429s on legitimate traffic from the session cap | until August 5; the cap dates from April 9, the first victim is undated | logs, August 5 |
| the empty French draft | since June, still there | nothing; I keep it |
| ops review not running on August 8, 9, 15, 16 | 26 days for the first one | me, counting transcripts on September 3 |
| 125 calls to tools that do not exist | 92 days | a CSV export, September 3 |
Two rows have no detector at all, only a person who happened to count. The four missing ops reviews were two closed-laptop weekends; nothing said so, because a run that does not start writes nothing, and nothing is the same colour as green. The invented tool names errored 125 times in front of a server that logs every call, and the log was never read for that question. Time to detection is the metric that tells you which of your checks are actually running. We do not lower the error rate. We shorten how long an error lives.
What survives at three weeks is the other half of that measure. For my scheduled runs: the full transcript, locally, with token counts and tool calls, so I can answer "what did it do, in what order" for any day since August. For the server: one row per call with keys and sizes, never values, and no fingerprint of which agent made it. For the June pipeline: nothing; the execution history is purged, and my memory of a green run is the only witness. Three systems, three answers to the same forensic question, and none of them can answer "was it the right action". Only the read-back can.
The debugger is also non-deterministic
The uncomfortable part of debugging a system that is allowed to be wrong is that the thing you debug it with is allowed to be wrong too.
My llms.txt annex read the wrong window. My first E2 verifier divided every French body by an English length of one and failed six correct drafts. And the glossary sentinel, before it was a script, was a cloud routine created on August 12 that rewrote its own checker at every run: a verifier that is a slightly different program each week cannot be debugged, because there is nothing stable to debug. It ran twice, both green. I replaced it on August 17, on that design argument alone, with a frozen script, 22 negative test cases, a positive control, under version control. Four reports since, all green on 170 pages, 89 to 166 seconds each. The state is the sitemap. The validator is the test file.
Freeze the verifier first. It is the only component whose correctness you can actually prove, and the only one nobody wants to write. On one article, 20 identical translation calls at temperature 0 on September 3 gave one distinct output; the eight deterministic checks I run on a translation caught all six faults I injected, and one of them, an altered URL, was caught by exactly one check. A judge would have shrugged. The status code would have said 200.
What is not built
Not shipped on our server, still: a heartbeat on scheduled runs; a token fingerprint in the usage row, so that "which agent" has an answer; an outcome flag next to the status code, so that a 200 that did the wrong thing shows on a dashboard; idempotency keys, version-bound writes, a plan-then-commit object; the three tickets above, open since August 5, unchanged at our August 28 commit. What is built: the error budget table, the incident registry, the frozen sentinel, and a read path I now know to measure rather than trust.
When your agent was wrong last time, what did you look at first, the prompt or the data? Genuinely curious which way the reflex goes, and how often the model turned out not to be the culprit. I will answer with the registry and the read-back numbers, cycle by cycle.
Top comments (17)
Data, or more precisely the read path. It took me longer than I would like to admit to get there, because my reflex used to be the prompt as well.
The reason it should not be the model is structural, and you name it yourself: every layer above the model is deterministic and checkable in minutes, while the prompt is the only layer where a fix cannot be verified. That is a good enough reason to order the search that way even if the model were the likelier culprit, which your registry suggests it is not.
Your read-after-write finding is the part I would want more people to sit with. A cache that bypasses for exactly one read, with a list call in between resetting it, is a defect that only becomes visible under agent traffic. A person writes and then looks. An agent writes, reads, lists, reads, and does it forty times a minute. The invariant the cache was built on was human pacing, and nothing in the design ever said so out loud. We have hit the same class of problem on shipment status endpoints, where a stale read is a vehicle going to the wrong place rather than a wrong paragraph, so the cost of the gap is easy to explain to people who do not care about caches.
Time to detection as the headline number instead of error rate is what I am taking away from this. Error rate describes the system. Detection time tells you which of your checks are actually running, and two rows having no detector at all is far more actionable than a percentage. Are you capturing that per channel as it happens now, or is it still reconstructed after the fact from transcripts?
Reconstructed, all of it. The "lived" column was computed on September 3 from three sources that were never meant to answer that question: the transcripts of my scheduled runs, one CSV export of the usage table, and commit dates. Nothing captures a detection timestamp as it happens. The only live detectors are Sentry on unhandled exceptions and the status code in the usage row, and neither carries an outcome. So the number is honest, but it is archaeology.
Live, per channel, would take the two items sitting on the "not built" list. A heartbeat, or rather an expected-runs table diffed every morning, because a run that does not start writes nothing and absence has to be produced by something. And an outcome flag next to the status code, so that a 200 that did the wrong thing is countable at all. With those two, time to detection stops being a spreadsheet and becomes a query: first detection minus first occurrence, grouped by what detected it.
Your framing is better than mine: the cache encoded human pacing as an invariant and never said so. Every TTL is a statement about who is reading. And the shipment example makes the cost legible in a way a wrong paragraph never will.
One question back: on your status endpoints, did the fix land on the read side, a version or generation the reader checks, or did the writer's own reads get a separate path?
Thanks, this one I will keep.
Read side, and not a version the reader checks, because in a tracking domain almost nobody is holding one.
Your exchange with Vinh is what makes me say that. Co-commit gives read your writes to a client carrying a version from its own write. On a shipment status endpoint the writer is a scan somewhere in the network, and the reader is a customer, or now an agent polling on its own schedule. Nothing joins them causally. "At least my version" is a sentence only a client holding a version can say, and here almost no client holds one, so the reader outside the chain that Vinh called mundane is the whole traffic profile rather than an edge.
So the read did not get fresher. It got honest about its age. The response carries the time of the underlying scan and the time the view it came from last advanced, kept as two fields on purpose. That lets a caller separate "no movement since 06:12 because nothing has scanned" from "I am handing you a copy and I do not know what has happened since". Those collapse to the same value in a single status field, and operationally they mean opposite things. The second one is what sends a vehicle to the wrong place.
The writer's own reads did get a separate path, but it is the smaller half of the work, and it exists for reconciliation jobs rather than for correctness at the edge.
One push on your table. The rule for a reader outside the chain is tolerated when no write preceded it, which is right for a dashboard and cannot be settled at the read path when the reader acts on what it gets back, because the cache has no idea what the caller is about to do with the answer. Putting the age in the payload moves that judgement to the only layer that knows. It also gives you the detection number on that channel for free, since first occurrence and first observation are both sitting in the response. Given you already log sizes per call, is there room in the usage row for that pair?
Possible, at least half of it. The copy's age I can stamp, since the cache is mine, and an age is not a value, so it fits in the usage row next to the sizes. The time of the underlying change is the other half, and that depends on what the service behind me returns.
Fair push on the table. The row says what I tolerate serving. Whether it is safe to act on is the caller's call, and the age is what lets the caller make it.
Worth testing the next time I run the read-back cycles. It would have replaced the probes: last run, I told the cache from the upstream by re-reading at 1, 3, 10, 30 and 61 seconds. A copy time on each response says which one answered, on the first read.
Honest about its age rather than fresher: that one I am taking away.
The half you are handing to upstream may be smaller than it looks, because three timestamps are getting collapsed into one.
When the world changed, when you last confirmed with the origin, and when you made the copy. Only the first belongs to the service behind you. The second is yours whatever that service returns, since you know when your fetch succeeded, and it is also the one that decides whether acting is safe. The thing you are protecting a caller from is a change that has already happened and that you have not seen yet. An unseen change can be at most as old as your last confirmation, and that number sits entirely on your side of the boundary.
If you want the change time as well, polling gives you an interval instead of a point. The first fetch that comes back different puts the change somewhere between that fetch and the one before it, so the width is your poll period rather than an unknown. That is worth saying out loud in the row, and it degrades cleanly on the day the service behind you starts stamping its own.
One caution on retiring the probes. A copy time written by your cache attributes staleness to your layer, which is the same boundary you drew for Vinh: the 4 in 50 is yours, the 2 in 50 is in a service you consume. If that service is also handing you a copy, your stamp records when you asked, not when the answer was made, so the ladder is still measuring something the stamp cannot see.
Is the read-back cadence fixed? The interval is only useful if the poll period is a number you can quote.
Not fixed, and on this path nothing polls at all, which I keep losing sight of because we poll in a lot of other places, for historical reasons. The cache goes back upstream on the first call after its 60 seconds, so two confirmations of the same key are at least a minute apart and otherwise set by traffic. No period to quote. A row could carry its own bounds instead, the previous confirmation and this one, but today the previous one expires with its copy.
Agreed on the three timestamps, and the second is already stored. The cache keeps the time the upstream answered and checks its 60 seconds against it. It is just never exposed. Confirmation and copy come apart around a write: the write and the bypassed read after it both go upstream and leave the older copy in place. That is the article's 50 out of 50: the list had seen the delete in 49 of them, and the get behind it was still handed the copy from before.
The caution stands. With that much polling around, what the service behind me returns can itself be a copy with an age of its own. The bound for acting is the age of my confirmation plus the age the answer already had when it reached me. The stamp gives the first term, and the ladder is still what measures the second. The 2 in 50 were bypassed reads served live: a copy time would have said "just fetched" on a wrong answer.
The probes stay. The stamp says whose copy answered, the ladder how far behind it was.
On the previous confirmation expiring with its copy: it does not have to. Carry it forward on write. When the cache refreshes a key, read the outgoing entry's confirmation time and store it in the new entry as the previous one before the value is overwritten. One extra timestamp per entry, the old value is still discarded, and the row gets the pair of bounds you wanted without retaining the thing you were trying not to retain.
Make that field nullable rather than defaulted. An entry created after an eviction has no predecessor, and a null there reads as "no bound available", where defaulting it to the TTL quietly claims the change happened inside the last sixty seconds, which is the one thing you do not know.
Given that nothing polls on this path, what the pair actually buys is that the width of the bound stops being unquotable and starts being reported per row. A hot key gets confirmations a minute apart, a quiet one gets them hours apart, and the row says which it was. I think that is better than the poll period you were reaching for, because a single period would have been an average across keys with very different traffic, and you would have been handing callers a number that was wrong in both directions at once.
On the 2 in 50, I think you have already put the stamp in its right place. Served live is where a copy time is most confidently wrong, since it reports maximum freshness on the one answer that was stale before it reached you. That is the argument for the ladder staying rather than for the stamp being weak: the stamp describes your layer, and only a re-read describes what your layer was handed.
Carry it forward, agreed, as long as the time outlives the copy. The copy has a 60-second lifetime and the refresh is the first call after it, so at that point there is no outgoing entry left to read. Two timestamps are not a value, so it stays inside the rule of the usage row: keys and sizes, never values.
Null rather than a default, yes. A default makes "no bound" the same colour as a bound, which is the article's problem with runs that never start: nothing is the same colour as green.
The width per row puts an error bar on the article's third question, how long did the error live, for my layer: a minute on a hot key, hours on a quiet one, where the copy expired long before the next confirmation. A single period would have been the cache's error rate, one number true of no key in particular. Per channel was the point, and per key is finer.
On the 2 in 50, same rule as the transcript in the article: the stamp is a witness, not evidence. It answers the attribution question, did the read-back read the world or a copy of it, and on those two it would have said not my copy, truthfully, about an answer that was already behind. So the ladder stays: the read path is still something to measure rather than trust.
The pair in the row goes where the fingerprint and the outcome flag already are: not built.
One correction and then I will get out of the way.
The carry forward does not fail because it is impossible. It fails because I scoped the timestamp to the value's lifetime. Keep a small map of key to timestamp that outlives the copy, written on every confirmation, and the previous confirmation is still there to read when the refresh arrives on a cold key. A key and a number, so it stays inside your rule.
That distinction matters more than it looks, because tying the two lifetimes together loses the bound on exactly the keys where it is widest. A hot key refreshes while its entry is still present and keeps its pair. A quiet key whose copy expired four hours ago is the one where the gap is interesting, and it is the one you would systematically drop. You would end up with per-row bounds that exist only for the rows that needed them least.
Agreed, and that is the better version. Give the time its own key and its own lifetime and it outlives the copy. A map of key to confirmation time is still keys and numbers, so the usage row keeps its rule.
You are right about where the loss would have landed, too. The quiet keys are where the bound is widest and where it matters most, and tying the timestamp to the value would have kept a bound only where it was narrowest. Two keys, not one. Still not built, but now buildable.
Thanks for the correction, and for staying in the way this long.
The two sequences differ in two places, so the
listis not isolated yet. A is create, read, delete, read; B is create, read, read, delete, list, read, which adds an interveninglistand also a second read before the delete - so "a list in between resets it" and "the bypass is spent by whichever read comes first after the write" both predict the 50 out of 50 you measured. The arm that separates them is create, read, read, delete, read with nolistin it at all: if that one is still stale, the extra read consumed the bypass andlistis innocent. I would want that before picking a fix, because the two readings want different code - one is "stop countinglistas a read that resets the bypass", the other needs a generation counter per token - and the 3 in 50 from sequence A is already saying the one-read bypass is not deterministic even in the shape it was built for.You're right on both counts, and the sentence in the article is wrong as written. S2 changed two things, and "a list in between resets it" makes the list special. It is not. The bypass is a one-shot flag set by a write and keyed on the token, spent by the next read on that token, whatever the tool. In S2 that read happened to be the list, and the get behind it hit the copy cached before the delete.
I ran your arm this morning, 50 cycles on the same test app, with create, read, delete, read as a same-day control. Create, read, read, delete, read, no list: stale 4 times out of 50, each still there at 1, 3, 10 and 30 seconds and gone at 61. Without any list the one-shot bypass itself misses about one time in twelve, and a miss with a warm cache costs 60 seconds. The control: stale 2 out of 50, gone by the first probe at 1.6 seconds. Those two are not the cache. The upstream had not caught up with the delete yet.
One boundary I should have drawn in the article: I run the MCP server, not every service it calls. The cache and the bypass are mine, so the 4 in 50 is mine to attribute; first suspect, a race between the write returning and the flag landing, and I read that path before touching anything. The sub-second window behind the 2 in 50 lives in a service I consume like any other client. I can name it and measure it, not close it, which is one more reason the verifier has to treat "not found" as unverified rather than as truth.
On the fix we agree. "Don't count list" was never the right code, and a flag with an 8 percent miss rate is not a guarantee. A write generation per token, stamped on the cache entries, is the read-side twin of the version-bound write on my list, and on my side of the boundary it has no window to lose.
Thanks, this is the arm I should have run.
The 4 in 50 is the one number a generation counter does not fix for free. Your own diagnosis of it is a race between the write returning and the flag landing, and a generation stamped by the same post-write path lands on the same schedule: a read arriving inside that window reads the old generation as the current one, the cached entry matches it, and the result looks fresh rather than unverified. Same window, quieter symptom.
What closes it is not generation-instead-of-flag, it is that the delete cannot return until its marker is durable, or that the marker comes out of the same commit as the write itself. Worth pinning down which of those your version is before the miss rate gets read as solved, because a marker that lands late still costs the full 60 seconds you measured, and that failure no longer has a one-shot flag to blame.
Agreed on the stamp. A generation written after the write is the flag with a longer name: same window, and it fails "fresh" where the flag failed "unverified".
So the marker has to come out of the commit itself. The write returns its committed version, the token carries that version forward, and the read no longer asks "what is the current generation" (that lookup is the stale one). It asks "is this cache entry at least the version my write handed me". Below it, or missing, means unverified: refetch. There is no post-write moment left to lag. It also means the delete has to return that version. Today it returns only the policy envelope, so that is part of the change.
Holding the delete until the marker is durable closes the window too, but it taxes every write and leans on a side channel next to the commit. Co-commit keeps one authority. That is the 4 in 50.
The 2 in 50 is not mine to close: a sub-second window in a service I consume and do not run. It stays the verifier's rule: not found is unverified, never truth.
Good catch. The stamp would have shipped looking solved.
Co-commit closes it for the writer, agreed. The gap I would still name is the reader who did not do the write: no token, so nothing to compare the entry against, and it falls back to asking what the current version is - which is the lookup you just established is the stale one. The guarantee is scoped to a causal chain starting at a write, and reads outside that chain are back where they started.
That may well be fine if every read on this path is downstream of a write in the same request. Worth checking that it actually holds, because the thing that breaks it is mundane rather than exotic: a second consumer polling the same key on its own schedule.
Not arguing against the change. "At least my version" is just a sentence only a client holding a version can say.
Agreed, and that is the right boundary to name. Co-commit gives read-your-writes to the client holding a version, nothing more. A reader outside that chain gets the 60-second copy, same as today, which is the first row of the table: tolerated when no write preceded it, never when one did, by the same token.
Whether tolerated is acceptable depends entirely on what the reader does with the copy. A poller refreshing a dashboard, fine. An agent that branches on the read is the 70 paragraphs: 25 stale reads in a row, an action on each. Where the reader acts, the copy is not ideal, and the fix there is invalidation on write, which only reaches my side of the boundary.
Lorca had the line for it: universal law is for lackeys, context is for kings. Which is what the table is for.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.