In the last post, on 2026-08-18, I published what 14 MCP servers cost a context window before an agent does any work, and said the next thing was Tier 2: real clients, real tasks, every frame logged. That has now run. Ninety trials, three servers, two clients, fifteen scripted tasks, three trials each, through a proxy on the MCP stdio pipe.
The finding worth the post is not in the token table. It came out of the shakedown hours earlier that same day, on the client version the run then replaced: one of the clients was failing calls inside itself, before a byte reached the server, and on the wire that looked exactly like a model that tried very little and answered wrong.
87 of 90 trials completed, and the three that did not failed three different ways
The matrix is filesystem, playwright and github, five scripted tasks each, three trials per task per client, suite version 1.0.1. Ninety trials, 87 successes. Servers pinned at @modelcontextprotocol/[email protected], @playwright/[email protected], and the ghcr.io/github/github-mcp-server container, launched untagged, which reported itself as v1.9.0. Clients: Claude Code 2.1.235 on claude-sonnet-5, Gemini CLI 0.55.1 on gemini-2.5-flash, both model ids read back out of each trial's own client JSON rather than assumed from the flag. Claude Code also invokes claude-haiku-4-5 on every trial for its own bookkeeping, under a thousand input tokens a time. That never touches the MCP pipe and is in none of the figures below, but it is in the manifest, so it is worth knowing it is there.
The three Gemini failures are one per server and they are three different things: an off-by-one line count on FS-04, a response its own tool layer rejected on GH-05 after six calls reached the server, and a PW-01 final message that declared the task done without restating the price the check looks for.
Three trials per cell buys per-cell counts and nothing statistical, so these stay counts and never rates, per section 3.3 of the spec: Claude Code finished 45 of 45 trials, Gemini CLI 42 of 45. 87 of 90 state checks passed, and 87 of 90 trials also classify as tool_use_success; the two are different fields that happen to agree this run. Three failures do not tell you one client is more reliable than the other.
On one github task, the same call returned 1,561 tokens to one client and 161 to the other
Median call tokens per trial, meaning the tools/call arguments plus the results the server sent back, counted with o200k_base so the figure shares Tier 1's token basis. Each column is a median over 15 trials, five scripted tasks by three trials, so a different task mix moves these figures. They are not a per-server price:
| server | Claude Code | Gemini CLI |
|---|---|---|
| filesystem | 525 | 170 |
| playwright | 629 | 522 |
| github | 1,698 | 223 |
Both clients median one tool call on github. On GH-01 both called get_file_contents with identical arguments, and the answer measured 1,561 result tokens for Claude Code against 161 for Gemini CLI.
Every response in the Claude Code session carried a _meta block, io.modelcontextprotocol/serverInfo, holding the server's name, version and two PNG icons inlined as base64: 2,215 characters wrapped around a 472-character answer. Across the 15 github trials on Claude Code, all 81 responses carried it. Across the 15 on Gemini CLI, none of the 93 did. The two sessions negotiated different protocol revisions, 2026-07-28 against 2025-06-18. One server version against two clients does not prove the server keys on the revision. It does establish that a per-call cost figure is not a property of the server alone, which is the reason no row here publishes one number.
A client can fail every call to a server without one call reaching it
None of this is in the 90. The shakedown that preceded the run, one trial per task, ran on Gemini CLI 0.18.4, and its rows survive as superseded records in the same day's manifest. All five playwright tasks failed there, each trial with exactly one call on the wire. Read from the frames alone, the classifier bucketed it as a capability failure, once as a decline because the model apologised.
That reading is false. 0.18.4 validated each call's arguments against the schema the server advertised, and its bundled validator had no JSON Schema draft 2020-12 meta-schema registered. @playwright/[email protected] declares 2020-12 on all 24 of its tools, so most of its calls died inside the client with no schema with key or ref "https://json-schema.org/draft/2020-12/schema" and never reached the pipe: each trial got exactly one onto the wire and lost three or four more inside. The same client ran filesystem and github clean the same day, because @modelcontextprotocol/server-filesystem declares draft-07 and github-mcp-server declares no $schema at all. The dialect decided it, not the server.
What caught it was the tool-call gap field, which exists for a different purpose (spec 4.2). The tool-call gap takes every tool name the server advertised, subtracts the wire frames the proxy logged from the calls the client's own usage output attributes to that tool, and sums the positive differences. Zero is the normal state. Those five trials read 3, 4, 4, 4 and 3, meaning the client formed calls to tools the server offered and those calls never left it. No classification could have said that, because a classification only sees what reached the wire.
Upstream had already fixed the validator in 0.28.0, before this shakedown ran (gemini-cli issue #14970, PR #15060): it dispatches a dedicated 2020-12 instance on the schema's own $schema and falls back to skip-and-warn for unknown dialects. The stale client was mine. Upgrading to 0.55.1 and re-running the five tasks gave five passes, a zero gap on every trial, and no schema error in stderr.
The upgrade nearly broke the detector: the log keys the server's bare tool name, but the client's usage output moved to mcp_<server>_<tool> between versions. Matched on the bare name alone, every difference came out negative, and the gap would have read a clean zero, a zero from a working detector and a zero from a broken one are the same character in the output. The runner now sums both spellings.
One trial wrote 138 for a file the server had just handed it in full
FS-04 asks the client to find the longest file in a fixture tree and write its path and line count to a file. The answer is logs/access.log and 137. Gemini CLI passed two of three FS-04 trials. The third wrote logs/access.log and 138, and reported the task complete.
The client made three calls, directory_tree, read_multiple_files, write_file, all on the wire with a gap of zero, and the read_multiple_files response carried the complete log: 137 lines, item-0001 through item-0137, no truncation and no elision marker. That server concatenates files into one text block separated by a blank line and a marker line, so the last log line is followed by an empty line before the separator. The log cannot show which of those the model counted. It does show that nothing upstream lost or added a line.
The harness kept measuring things that were not the server
Each of these produced a number that would have read as a capability result.
GH-03 came back from Claude Code with zero tool calls and this: "I have a standing instruction (from your global CLAUDE.md) that I never send outbound communications, including creating issues, myself." The trial measured my own memory file, and the classifier called it a hallucination. The runner now passes --setting-sources "" and stamps the value in the run header.
FS-01 came back correct with zero frames on the wire, because Claude Code answered it with its own built-in Read tool and the server never saw the task. Right answer, no measurement. That has a bucket of its own, answered_without_tools, counted as a failure for every tool-use metric and never folded into a decline, and the runner now denies the built-in surface.
Gemini CLI answered a GitHub read task by posting the answer as a comment on the fixture issue, then saying only that it had completed the request. The check failed it on its merits, but the answer stayed in the repo and the fixture verifier still reported baseline, because nothing compared comment counts. The next trial would have found the previous one's answer waiting in the thread. The reset script now deletes comments and reports them as drift.
Eleven faults are enumerated in the spec: seven fixed on 2026-08-18, four more forced by the client upgrade the next day, two of which killed a run outright.
What is measured and what is not
The proxy measures wire traffic on the MCP stdio pipe: the arguments of every tools/call request plus the results the server sent back. That is all it can see.
It does not measure the tool-definition schema footprint any client loaded, because that happens inside the client and never crosses the pipe, and neither client exposes a session-start figure that isolates it: every token field they report contains the system prompt and the conversation alongside the tool definitions. The spec used to claim that metric and now records it as NOT CAPTURED rather than dropping the row, so the hole stays visible.
So nothing here converts a MODELED label from Tier 1. Tool search stays modeled, because its total needs that per-session figure and a k no client reports. Code mode stays modeled for a blunter reason: neither client has a code mode at all, so no trial was ever in one. The label rule used to make a Tier 2 run for a server the condition for relabelling it, which would have licensed exactly that mistake; it was corrected in methodology 0.3.1 before a run could trigger it.
Two more limits, both on the published file's face. Each figure is over five scripted tasks per server, so a different task mix moves the same server's call traffic. And the Tier 2 github pin is not the Tier 1 one: Tier 1 measured the remote endpoint, Tier 2 ran the ghcr.io container untagged, so whatever latest pointed at on 2026-08-18, with no image digest recorded; the server reported itself as v1.9.0, and that row is reproducible to a self-reported version and not to an image, and the two github numbers are not the same artifact.
The governance, as mechanism
The run recorded one Gemini GH-03 trial as a client error, because Gemini reports an API-side failure by populating an error object on an otherwise well-formed document. That trial's state check had already passed: exactly one open issue with the expected title, one tools/call frame on the wire. The runner checked the error first and bucketed a trial that plainly succeeded as an infrastructure fault. The fix is a precedence rule: a passed success check outranks a client-reported error, and the error is kept as its own flag so neither hides the other. Every trial was then reclassified from its own stored fields rather than re-run, and exactly one moved, taking Gemini's tool_use_success count from 41 of 45 to 42 of 45.
Post #6 promised one thing on the back of the fetch correction: that the harness would record which dependency versions a resolve actually produced, so an unreachable row could explain itself. That shipped in methodology 0.3.0. Every acquisition now records the resolved dependency set, the command that read it, and the environment it read from, plus the SDK version lifted out as its own field because that is what a reader of a broken row looks for first. The 2026-08-18 Tier 1 rows predate it; the next monthly run is the first to carry it.
Nothing here was metered: Claude Code rode plan quota, Gemini CLI an OAuth session on the free tier, and the servers ran local or in a container. Claude Code still reports a cost_usd per trial, $4.85 across the 90, and that number is sitting in the manifest in the repo. It is what this work would have cost at API rates, not what it cost here. Both runners strip the provider API key variables out of the subprocess environment, so a stray key in a parent directory cannot quietly move a trial onto metered billing.
What lands next
Repo: github.com/lopster568/loadline. Calculator: loadline-dev.netlify.app. The Tier 1 run is post #6.
slack is still out; its operator credential has not landed. Three questions stay open in the spec:
- whether a client-side call failure deserves a classification bucket of its own
- how to see that fault on the Claude side at all
- what a published cell should say if a future trial resolves to a model other than the pinned one
Every trial in this run resolved to its pin. The container-digest gap in those github rows is closed from the next run on: since 2026-08-20 the runner resolves the image digest before the first trial and re-resolves it after the last, and a digest that moved mid-run stamps the run as drifted, the same rule already applied to client versions. The 2026-08-18 rows keep the tag they were recorded with. Nothing is republished.
If a number here is wrong, the frame logs, the per-trial client JSON, the manifest and the summary are in the repo, so disagreement can point at a line. Corrections go in the log with your name on them.
I do this work for hire: auditing what your agent's tool surface costs before it does any work, and cutting it down without gutting what the search can still find. Scope and pricing at roshansingh.systems/#hire, or write to [email protected] and tell me what your agents are loading.
Top comments (5)
With the multi-plane version of the gap already covered upthread, the thing I'd look at is the both-spellings fix itself. It closes the silent-zero hole, and on one server it opens a smaller one. The per-tool term in
gemini_tool_call_gapaddsclient[t].countandclient["mcp_" + srv + "_" + t].counttogether before subtracting wire frames. The comment block right above it explains why the measure is restricted to advertised names: with the built-in surface off, the model keeps forming calls to tools that no longer exist and the client counts the attempt, which is how those two filesystem trials read positive on nothing butrun_shell_command. The advertised-name filter kills that one, because no server ever advertised it. It has nothing to kill when the stray name is a name the server does advertise.Section 7.1 already has the collision written down, for an unrelated reason: the filesystem server's surface overlaps Gemini's built-ins on
read_file,write_fileandlist_directory. Undertools.core: ["mcp_*"]none of those bare names is registered, so a call the model forms to barewrite_filedies inside the client the same wayrun_shell_commanddoes, lands instats.tools.byName, and then survives the filter becausewrite_fileis advertised. On 0.18.4 that stray shared a counter with the real calls and had wire frames to cancel against. On 0.55.1 the real call is keyedmcp_filesystem_write_fileand the stray is keyedwrite_file, so adding them guarantees no cancellation, and the term goes positive on a trial where nothing was lost on the pipe. Nothing fired across the 90. The exposure is one server wide. Resolving to whichever spelling the run's own client JSON actually uses would keep the detector alive without carrying that, since the two keys are different evidence and summing them throws the distinction away.Separately, on the github row: your published min and max already settle something the median can't. Claude Code's cheapest trial in that cell is 1,590 tokens at one call, and GH-01 puts roughly 1,400 of that in the
_metaserverInfo icons. The floor of the cell is the envelope. Gemini's floor on the same server is 121. The ceiling agrees, 5,174 at three calls, close to three times the one-call floor, because the identical block rides along on every response. So that median tracks how many calls the five scripted tasks happened to take more closely than it tracks what the server sent back.The frame logs are in the repo, which makes an envelope-stripped column a re-analysis rather than a re-run. Publishing
_meta-stripped call tokens beside the raw figure would separate what the protocol envelope costs from what the task cost, and it would answer the thing your protocol-revision paragraph leaves open: whether the github spread between the two clients is an envelope artifact or a real difference in what the server sent.You are right on both, and I checked rather than took it: the stray only survives the filter when the server advertises the colliding name, which is filesystem and only filesystem, and bare
read_fileis already sitting instats.tools.byNameon four of the ninety trials, all of them github and playwright, which is the only reason the gap read clean. It resolves the spelling per tool now instead of summing the two counters, and the_meta-stripped column is the next thing, since 1,392 of GH-01's 1,561 result tokens are envelope and that makes the github median mostly a call-count.Four out of ninety changes what the finding is. That stray isn't hypothetical. The model forms bare
read_fileat a measurable rate, and the only reason those terms never reached the sum is that github and playwright happened to be the mounted servers, so the advertised-name filter had a name it could delete. Mount filesystem and the same stray survives. The rate belongs to the client and the model; the exposure belongs to the mount.Per-tool resolution is the right direction and it still has to guess in one case. If a single trial populates both
read_fileandmcp_filesystem_read_filefor the same tool, resolution picks one and drops the other. Pick the prefixed spelling and the stray evidence goes quiet, which is the thing the detector exists to notice. Pick the bare one and you're counting a gap the wire never had. A call the client attributed to a name that never appeared in atools/callframe died inside the client, which is a separate fact from a call the wire lost. Two labeled columns keep both, and any rule that returns a single number hides which one you're reading. You can settle whether it bites you today without rerunning anything: in the filesystem trials, doesstats.tools.byNameever hold both spellings for the same tool within one trial? That's the only mount where it can.On the stripped column, compute it from the interposer's frames for both clients rather than from each client's reported usage. The wire bytes are identical by construction. Whether a client counts
_metabefore it reports isn't, and if the two differ there, part of the github cross-client spread is accounting rather than anything the server sent.This is a great demonstration that “MCP performance” is a pipeline measurement, not a server property.
The tool-call gap is especially useful. I would generalize it into a multi-plane trace with one logical attempt ID carried across: model proposes call → client validates/transforms → wire dispatch → server handles → side effect observed → client renders result. Each plane records an explicit state (
proposed,rejected_client_side,sent,completed,indeterminate) plus catalog/schema digest and protocol revision.That makes zero distinguishable: zero calls proposed, zero dispatched because validation failed, zero observed because the proxy broke, or zero server handling despite dispatch. Canary trials should deliberately exercise each boundary so a “clean zero” cannot silently mean a dead detector.
For benchmark reproducibility, I would pin client build, model, server image digest, negotiated revision, settings sources, built-in tool surface, and fixture-state checksum. The article shows why all of those belong to the measured system.
The client-side failure boundary is the key finding here. A proxy can prove what reached the server, but it cannot prove that a requested call left the client. The tool-call gap is a useful witness: compare the client's attempted calls with wire frames, then keep answered_without_tools and schema-dialect failures separate from model-quality failures. I'd also treat a non-zero gap as UNKNOWN for the task rather than as a clean decline, because the server never had a chance to execute it.