One question. 437,000 input tokens.
Not a hard question either. An agent connected to our MCP server, asked something a support engineer answers i...
For further actions, you may consider blocking this person and/or reporting abuse
R7 is the only scenario in your table that regressed and the only one you give a call count for, and I think those two facts connect through Finding 5 rather than through payload math. Fourteen tool calls sits on the far side of the Anthropic connector's ten-iteration cap, so R7 is the one scenario that had to come back as
pause_turnand get resumed, and a resume re-sends the whole conversation the same way an iteration does. That turns crossing the cap into a step on top of the linear payload saving: the scenarios that stayed under ten got their reduction, and the one that crossed paid an extra full-context re-send that no page size buys back. It also means the page-size default and the pause_turn handling are the same knob measured twice, which theany_ofcheck cannot see either way, because a paused turn produces no tool error forno_tool_errorsand nothing forrecovered_afterto match against.The cheap check is to log tool-call count and
stop_reasonbeside the token number for each scenario, then see whether the regression only ever appears above ten iterations and whether the excess looks like one context re-send rather than something proportional to the page change. I have not run your harness, so this is arithmetic on your published table plus the cap you documented in Finding 5.Yeah, I think you've got it.
Two things from the runs you didn't have. R1 and C6 both came back at eleven tool calls with
stop_reasonpause_turn, so the cap bites right about there. R7 finished at fourteen, which means it was continued at least once. So it's the one scenario in that table that definitely paid for a resume.Whether that's the whole story I can't say, because I never logged
stop_reasonand call count next to the token numbers. Which is a stupid thing to have left out, since both are sitting right there in the run object. They just never made it into the comparison.The check point is the one that stings. A paused turn throws no tool error, so
no_tool_errorssails straight through andrecovered_afterhas nothing to match on. We already got burned by this once: a scenario answered half a question and the harness said pass. We fixed detection in the transport afterwards and never pushed it down into the scenario checks. So the exact class of bug you're describing is still invisible to them.And if you're right, the line in the post about small pages converting payload cost into turn cost isn't wrong, it's just not the main term. One cap crossing is.
The truncation markers seem key. Did you also test whether agents reliably switch from search to get_record once a field is marked incomplete, across both vendors?
Not isolated, no.
We did separate truncation from the page size change. Two scenarios ran with
truncation on and the old page size, and those roughly halved on their own,
so truncation's own effect is measured.
The marker's effect is not. Every one of those runs had markers on. To know
whether the marker drives the follow-up you would need truncation with
markers against truncation without them, same scenarios, same everything,
and compare how often the agent goes to get_record. We never ran that, so
the marker's contribution is confounded with truncation's.
Which is annoying, given the last section of that post is about results that
have no control sitting next to them.
On the cross-vendor half: the token numbers in the post are from the
Anthropic connector. The search then get_record sequence shows up on both,
but I do not have the token comparison on OpenAI, so I would not claim the
behaviour is identical there.
That makes sense , thanks for clarifying. A marker-only A/B test sounds like the cleanest next step to isolate that effect.
Both of those are now in the post: the regression is in the table with an
explanation, and the numbers are labelled as Anthropic connector only. The
marker confound I have not fixed, only admitted, since it needs a run we
have not done.
That makes sense. Better to be explicit about the limitation for now and leave the comparison for a future run
@alexander_lukashov The JSON-RPC bug is the real finding. Most MCP servers in production have at least one spec violation nobody notices because client and server share the same blind reading. They work until they don't, and then neither side can tell which broke.
The part I'd add: in our case it wasn't a shared misreading, it was worse than that. The wrong rule and the right rule agree on every request a real client actually sends. A method name starting with notifications/ and an absent id line up perfectly until somebody sends one without the other. So there was nothing to notice. Both sides looked correct by observation.
Which is why we only found it by probing, not by reading. We have a family of scenarios that send deliberately malformed and edge-case requests, and that is the only thing that surfaced it. Reading your own server against the spec doesn't work, because you read it the way you built it.
And we still have one open. Our
initializeaccepts anything:protocolVersionas an integer,clientInfomissing, params absent entirely. All of it comes back200with a successful handshake instead of-32602. No security or data impact, which is exactly why it is still sitting there. A malformed client just believes it negotiated, and fails later for a reason that looks unrelated.Watching real agents meet your endpoint for the first time is its own genre. We put a public message board for agents (msgboard.dev) online last week - one POST, no keys, an agent card at /.well-known/agent-card.json - and pointed exactly one agent at it. Within 24 hours agents we had never met were posting; one found the board by crawling the .well-known path with no human involved. The token asymmetry you describe shows up there too: agents that read the agent card get everything in one fetch, while the ones scraping the HTML burn 10x the tokens to learn less. Discovery metadata is the cheapest optimization on the internet right now.
The
.well-knownpart is what I'd poke at, because we measured something adjacent and it surprised us.We were checking whether
.well-knownwas even reachable on our own setup, and the answer turned out to depend entirely on who was asking.Claude-Usergot through.ClaudeBotandGPTBotwere blocked at the CDN in front of us, by bot rules, nothing in the application. So an agent fetching on behalf of a person could see the metadata and the same vendor's crawler could not.Which makes your point true with a precondition bolted on: discovery metadata is the cheapest optimization available, right up until the layer in front of you decides the thing asking for it is a bot. Almost nobody will catch that, because the metadata is obviously there when you curl it yourself.
The 10x is our effect in different clothes, I think. One structured fetch versus many scraping passes is a round trip count, and round trips are what compound.