Somewhere between June 3 and September 2, 2026, an agent asked our production MCP server for a tool called GBContent.getItems(sectionId, opts, onOk, onErr). Parentheses, parameter names, callbacks, the whole JavaScript signature, sent as the name of a tool. The server has never had a tool by that name. It answered with an error, the way it answered the 124 other calls to 33 other tools that do not exist: get_posts, push_send, skills/cms, tools/list as a tool, and a file path, /tmp/gb_args.json. Five apps, 125 calls, 92 days, one hundred percent errors. Somewhere, a transcript said "let me try a different approach" and moved on.
That is not in any demo, and it is not in most of the "lessons from production" threads either, because the person writing the thread usually built the agent. I run the other side. One line of context so you know where I stand: I run engineering at GoodBarber, an app platform, and we operate a production MCP server that customers' agents call all day. My own agents run on a laptop, not in production, and the laptop closes on weekends. Everything below is counted from systems I operate myself, with the window and the date in the sentence. No customer names, no app ids, aggregates only; the volumes are rounded on purpose, the ratios are exact.
The fleet, dated
Here is what "agents in production" means from where I sit, as of September 2, 2026.
| what | where | engine | model | cadence | who presses | runs |
|---|---|---|---|---|---|---|
| public MCP server | prod | our code, customers' agents | theirs, never ours | continuous | the customer | ~100k calls, 100+ apps (Jun 3–Sep 2) |
| content agents | laptop → prod CMS | Claude Code | yes | on demand | me (CMS approval) | not counted |
| noon ops review | laptop | scheduled Claude Code | yes | daily 12:04 | Nagios/BuildBot → DM | 30 (Aug 1–Sep 2) |
| two code reviews | laptop | scheduled Claude Code | yes | every morning | git diff → me | 23 & 29 |
| glossary sentinel | laptop | frozen Python | report only | Mondays | 170 pages → report | 4 green (since Aug 17) |
| SEO checkpoints | laptop | one-shot tasks | yes | J+21/28/42 | Search Console → report | 4 run / 7 sched. |
| product AI | prod | RAG + back-office | yes | continuous | the customer | dozens of apps |
Seven lines. Two are production, and the model in the loop there is the customer's or the product's, never mine; the five others run on a laptop. One line has no model in the check itself, and it used to: the glossary sentinel started life on August 12 as a cloud routine that rewrote its own checker every week, ran twice, both green, and was replaced on August 17 by a script with 22 negative test cases. Not because it drifted; because a checker that is a slightly different program each week cannot be debugged, and I stopped it before it had a chance to prove that. Four reports since, all green, 170 pages each, 89 to 166 seconds.
The line that matters most is the first one, because it is the one I see least.
What a server operator sees, and what it cannot
Every tool call on our MCP server writes one row: app, tool name, client IP, status, duration, request and response sizes, and the keys of the arguments, never their values. No session id, because since August 10 the transport is stateless and there is no session. No token fingerprint, no user agent, no protocol version. That table is the whole of my visibility into other people's agents. I see the requests. I never see the conversation.
From that table, June 3 to September 2, 2026, our two test apps excluded:
- Close to a hundred thousand calls from over a hundred apps and close to a thousand client IPs. The median day grew tenfold, from a few hundred calls in June to a couple of thousand in the first days of September.
- 62.8 percent of calls are writes. The top tool is
cms_create_article_paragraph, thencms_update_event. Agents are not browsing our customers' apps. They are filling them. - Concentration: the top five apps hold 55.1 percent of the calls, the median app made a few dozen calls in three months, and more than one app in five was used on a single day and never again.
- Several thousand push calls. A push notification has no undo. Agents sent, or scheduled, thousands of them through us in 92 days, most of them through a single tool.
- Error rate 2.9 percent, 30 timeouts in close to a hundred thousand calls, and the 125 calls to tools that do not exist.
Now the number I actually wanted. In the write-safety piece I argued that a write is not done until a read confirms it, and our server tells every agent so: each write returns _mcp_policy.verification_required: true with the tools to read back with. Do they? I grouped calls into sessions by app and client IP with a ten-minute gap (a heuristic, the only key the table allows), and asked how many writes were followed by a read of the same family within 120 seconds.
41.0 percent, the six fire-and-forget push tools left out of the denominator, median delay 18.9 seconds. For content, 41.5 percent. For the shop, 8.3 percent. On the stateless transport alone, August 12 to September 2: 42.2 percent.
So the honest sentence is: our server asks for a read-back on every write, and on the wire, three writes out of five never get one. The policy is advice. The clients decide. And I cannot tell which client, because the table does not know: that column, the sha256 of the token the server already computes for rate limiting, is the first thing I would add, and it is not shipped.
Silence has three shapes
My own scheduled agents run on my laptop and leave their transcripts there, so those I can count exactly. From August 1 to September 2, 2026, counters only:
- The noon ops review ran on 29 of 33 days. The four missing days are August 8, 9, 15 and 16: two weekends, a closed laptop. Nothing announced the absence; a run that does not start writes nothing, and nothing is the same colour as green.
- One code review, weekdays only, ran 23 of 23. The other, daily, 28 of 33: the same weekends, plus August 2.
- The sentinel ran 3 of 3 Mondays.
Three shapes of silence, then. Absent: the run never started, and the only witness is a gap in a folder. Green and empty: in June, a translation pipeline of mine reported success on a French draft with a title and no body; I remember the run as green, its history is purged, and the article is still empty in the test app. Unread: a report nobody opens is indistinguishable from a report nobody sent. The fix for the first shape is boring and I do not have it yet: a heartbeat on every run, absence as the alert.
The debugger fails too. My llms.txt piece in August was built on a forensic read of 121 days of logs; the first version of its annex read 8 days and reported it as the whole window. The logs were fine. My analysis tool was wrong, and I caught it before publishing only because a number looked too round. In June, agents patched translations on seven live blogs, live, no draft; a diff afterwards found 46 damaged paragraphs. The agents were not lying. They were reporting. Nobody had asked the right question.
The bill nobody itemizes
Per run, from the transcripts, medians over August:
| agent | turns | tool calls | uncached input tokens | cache-read tokens | wall time |
|---|---|---|---|---|---|
| noon ops review | 25 | 14 | 49 | 1,529,904 | about 3 minutes |
| code review, workspace | 122 | 71 | 336 | 14,689,395 | about 23 minutes |
| code review, AI workspace | 102 | 56 | 240 | 10,637,775 | about 21 minutes |
| glossary sentinel task (runs the script, comments the report) | 13 | 7 | 26 | 820,312 | about 8 minutes |
Read those columns together. A scheduled agent costs a few dozen uncached tokens a run; the rest of its prompt, one to fifteen million tokens, is the cached prefix re-read at every turn, and the code reviews re-read it a hundred times. The tokens are cheap at cache prices and the runs are on a subscription, so the bill I feel is not that one. It is the other column: the reports are read by a human. On a good week that is five minutes a day, six days a week, about half an hour, and most of it is spent confirming that nothing happened. On a bad week I have never counted.
The line item no calculator shows is maintenance, and the server's git log itemizes it better than I could. Since going live on April 10, 2026: a session cap that starved legitimate traffic with 429s until August 5, then became moot on August 10 when the transport went stateless; a rate limiter promised for "v1.1" on April 9 that arrived on August 11; two Redis incidents on July 29 and 31; a schema change made for a directory review on June 5 that silently hid one tool from some clients for 17 days, because those clients drop any tool whose schema has a oneOf at the root; an access token lifetime set to 300 seconds "just in case" on April 10 that logged every client out every five minutes until April 14. None of it is a model being wrong. All of it is the agent surface being a production system.
Who holds the button
Write down where the human is, or the demo will put one wherever it looks good.
- Content agents: every draft lands as
nonpreton our CMS and stays there until I approve it. Nothing they write reaches a reader without a human click. - The ops review and the code reviews: read-only by construction. They post to a person, never to a channel, never to a system.
- The server: no mandatory approval flow. The customer's client decides how much autonomy the agent gets; the server advises (
verification_required, a tool plan,max_retries: 1) and blocks nothing except what the credential cannot reach. A strict mode that refuses a write unless a read came first exists and is off by default. - Push: the six push tools return no verification hint, because there is nothing to read back. My own standing rule is
send: "at", nevernow. Thousands of pushes went through in 92 days and I have no idea how many were scheduled versus immediate. That is a column I could add, and have not.
The honest list
- No heartbeat on my scheduled runs, and a laptop is not a scheduler. Absent looks like green.
- No token fingerprint in the usage table, so read-back coverage is a heuristic and per-client numbers do not exist.
- The usage row is inserted on the hot path, before the response goes out, and is best effort: a database hiccup loses the row silently. No alerting sits on that table. No traces, no spans.
- The server's read-back bypasses the cache only for the read that immediately follows a write, by the same token. On September 3, in 50 cycles on a test app, a get placed after a delete and a list still returned the deleted object 50 times out of 50, for 61 seconds. Three immediate read-backs out of 50 were stale too.
- Not shipped, still: idempotency keys, version-bound writes, a plan-then-commit object, semantic span attributes. Prescribed in August, unchanged at our August 28 commit.
- Ticket B, C, D from my August audit, all still open: a delete returns only the policy envelope;
cms_create_articledefaults topublished; the verification hints on a create listcms_get_eventbecause they are the first three read tools in alphabetical order. - 34 tool names that do not exist were called 125 times. I did not know until I counted.
None of this shows in a demo because a demo is one run, watched, on a happy path. Production is the runs nobody watched, and the table that says what they did.
How do you know your scheduled agent ran this morning? Not that it was right: that it ran. Genuinely curious. And if you operate a tool surface for other people's agents: what share of their writes gets read back, and do you know which client is which? I will trade numbers, counting scripts included.
Top comments (0)