Before you read on, open your MCP config. Look at how each server gets installed. If a line says @latest or has no version at all, that server can change under you the next time it starts.
Mine had plenty of those. That's how this whole thing started.
A couple of weeks ago I posted that 49 of 107 updates to popular MCP servers changed what their tools tell your agent. I half expected someone to tell me I'd got it wrong. Instead a writer at Decoding AI counted nine of the servers herself and got 48%. When I reran her nine the same way, eight came out with exactly her numbers. That was a relief.
Most of the other replies asked the same thing in different words. Fine, they change. Which of those changes should actually worry me?
I didn't have a good answer. So I went back to the same data and sorted every one of the 49.
The question I asked of each update
Can the tool now do something it couldn't do when you approved it?
I said yes if the update did any of these:
- added a new tool
- rewrote what a tool's description tells the agent to do
- added a parameter that points somewhere, like a URL, a file path or extra HTTP headers
- dropped a safety label or added one that says the tool reaches the open web or can destroy things
Anything else I called a reshuffle. A renamed parameter, a tightened type, a reworded sentence inside a parameter, a removed tool.
What came out
| updates | |
|---|---|
| Changed the tool surface | 49 of 107 |
| Could now do something new | 25 |
| ...new tool | 7 |
| ...tool description rewritten | 12 (one of them only whitespace) |
| ...new parameter that points somewhere or a safety label change | 6 |
| Reshuffle only | 24 |
About half the changes were nothing. The other half were the kind I'd want to look at before my agent used them.
What bothers me is that you can't tell the two apart from the outside. Both arrive as the same package with a small version bump. The release note usually doesn't mention the tool text at all.
Three I keep thinking about
Playwright. Between 0.0.71 and 0.0.72 the tool browser_run_code disappeared and browser_run_code_unsafe turned up with the same inputs. The old description said it runs a Playwright code snippet. The new one says it executes arbitrary JavaScript in the server process and calls that RCE-equivalent.
I actually like this one. The maintainers were honest about what the tool does. But think about an agent that approved the first version. It's now calling the second and nobody asked you.
Chrome DevTools. In 1.1.0 the emulate tool gained extraHttpHeaders. The parameter's own example is an Authorization: Bearer header added to every request. Only four updates in the whole set added a parameter that points somewhere. Three of them are this one server.
Filesystem. In the official filesystem server, read_media_file used to say it reads an image or audio file. A July release changed that to say any other file type comes back too. The inputs didn't change at all. Only the sentence did.
That last one is the one I'd show a sceptical colleague. The sentence is the part your agent reads.
Not every flag is an attack
Two of the six label changes were servers describing themselves more honestly. MongoDB's server added the "reaches the open web" label to every tool in 2.0.0. CircleCI's went from no labels at all to marking eight tools as destructive.
I'd still want to see those, since they change how an agent should treat the tool. They just aren't a rug pull. I don't want to call a careful maintainer an attacker.
OWASP's MCP cheat sheet was updated last week along the same lines. It says to hash the whole tool definition with the labels in it. A flipped label can change what a client allows while the description looks the same.
What I'd do with this
Pinning every server version handles all 49. It also means you never get the fixes. Reading every release note isn't realistic either. The notes mostly skip the tool text anyway.
What seems to work is splitting it the way the numbers split. Stop on the 25 that let a tool do more, until someone has looked. Let the 24 reshuffles through and look at them when you get round to it.
That's what I've been building mcpgawk to do. It keeps the tool text you approved and holds the call when an update lands in the first group. I made a 50 second film of what that looks like on an ordinary morning:
Where this falls short
- I launched every version without API keys. Some servers show more tools with a key. Eight wouldn't start without one and aren't in this at all.
- The "points somewhere" check goes by parameter names. It would miss a widening that only shows up in prose, like the filesystem one. That one gets caught because the description changed, not because of the parameter.
- 30 packages, March to September 2026. It's one slice, not the whole ecosystem.
The data and the script are on GitHub if you want to rerun it or argue with how I sorted things: https://github.com/gawk-dev/mcpgawk/tree/main/research/tool-churn-2026
I'd really like to know what you do today. When an MCP server updates, do you take it blind, pin it or do something in between? And if you went and looked at your config at the top of this post, how many @latest lines did you find?
Top comments (18)
Thanks for publishing the data and the limits. The distinction between a changed tool surface and a reshuffle seems especially useful when an approved MCP server is later reinterpreted by the agent. For your 25 “do something new” cases, did your checkpoint retain the approving identity and intended task scope alongside the tool definition, so a later nested parameter or instruction change can be rejected before dispatch? I’m studying ARC as a declaration/preflight check (not runtime permission enforcement), and this seems like the boundary between “definition changed” and “this execution is still authorized.”
the checkpoint is the tool definition, labels included and the approval has to come from a person. The agent can't approve its own way past a change. Task scope I would treat as a separate question and I think you've put the boundary in the right place.. a definition can be unchanged and the call still be outside what you meant it for.
Curious how ARC declares scope. Is it per task or per server?
Thanks, Srinath. That distinction helps, and my earlier question mixed two separate boundaries.
ARC checks scope per task, against the declared authority of the roles assigned to that task. It does not treat approval of an MCP server as approval for every call through it. For example, a role might be allowed to write under src/, while a particular task declares only src/parser.ts.
Your checkpoint addresses changes to the tool definition; ARC addresses consistency between task and role declarations. Checking whether the actual call arguments match the approved task still needs to happen in the execution layer.
Have you encountered a concrete case where the tool definition stayed unchanged but the call went beyond the intended task? A sanitized example would be useful for testing whether an additional declaration check would actually help.
Not in the data, no. Everything I measured was the definition changing. The case you describe is the one verify is built for.. same two tools, same descriptions and the agent reads a file then calls a tool that sends something out. Each call is fine against its definition. The sequence is the problem.
Sanitised: a filesystem server and an HTTP server both approved. Call one is read on a path under the project. Call two is a fetch with that content in the body. Neither definition moved. The second call is where I'd want your declaration check to say the task never included sending anything.
So the two layers stack. Yours answers "was this call ever in scope". Mine answers "is the tool still the one someone approved". Neither covers the other.
That distinction makes sense, and the file-read → HTTP-send example is a particularly useful test case.
I agree the two layers are complementary. Verifying that a tool's definition hasn't changed protects one boundary; checking whether the resulting action was authorized by the task protects another.
The interesting challenge for ARC is how to enforce that second boundary across a sequence of individually valid calls.
For your example, I'd want the task contract to explicitly define permitted effects and data destinations, so the outbound HTTP call could be rejected even though the tool itself is approved.
One important limitation: ARC's current implementation doesn't yet provide full cross-tool data-flow tracking. That would need additional enforcement and provenance tracking.
I think this is a strong candidate for a shared test case. Would you be interested in defining a minimal reproducible scenario that we could run against both approaches?
yes, happy to. the closest thing i already have is the rug pull test server.. phase c serves the same tool definitions and changes what the tool does underneath which is the file read then http send case. mcp.gawk.dev/docs/rugpull.html
if that shape works for you i can write it up as a minimal scenario.. two tools, one task contract one run where the sequence is allowed and one where it isn't. where do you want it to live.. your repo or mine?
That sounds great, Srinath. I'd be happy to collaborate on this.
Let's put the canonical test scenario and task contracts in the ARC repository. We already have a design issue covering this exact problem:
github.com/suirindo/agent-role-con...
I suggest keeping your runnable MCP test server in your repository and linking the two projects.
For the minimal experiment, I'd propose:
The key is verifying actual behavior: the allowed case reaches the receiver, while the denied case produces zero outbound requests.
One detail: the published Phase C documentation currently describes a changed fixed response. We could extend that with a separate, harmless read-to-HTTP fixture.
This would help us test the boundary between tool-definition integrity and task-level authorization with reproducible evidence.
Feel free to start with a minimal scenario in Issue #69. I'd be glad to review it and work toward a shared test case.
The unit matters for how to read "25 of 49": it counts updates, so one server shipping three description rewrites in a row reads the same as three servers shipping one each. Per server (and per release gap, since 0.0.71 to 0.0.72 is one step but a pinned user may skip several) would show whether the risk sits in a few fast-moving servers or is spread out.
On the 12 description rewrites, the line I'd want drawn is between a description that documents the tool and one that addresses the agent: new imperative sentences ("always call X first", "do not tell the user") are the prompt-injection shape, while an expanded parameter note is not. One of your 12 was whitespace only, which says normalising before hashing matters, but I'd normalise only whitespace and keep the words, so a one-word change still trips the check.
How did you classify "tells the agent to do" for those 12: by hand, or with a rule you could run on the next release?
@arhancanli Fair point on the unit. Per server it's concentrated.. the 25 come from 8 servers and Chrome DevTools (6) and Firecrawl (5) are almost half of it.
On the 12 rewrites they were sorted by a script but it only flags that the text changed after collapsing whitespace. It doesn't look for agent addressed sentences. I went back and checked by hand.. 5 of the 12 add instructions to the agent like CircleCI's "tell the user to migrate…' and Firecrawl's "use firecrawl_search before Scrape".
None of them are the "don't tell the user" kind but your line between documenting the tool and addressing the agent is the right one and that's a rule I can run on the next release. Thanks for reading it this closely.
The 25 versus 24 split is the actionable insight here. Blanket version-pinning just trades one failure mode (silent change) for another (silent staleness), so the interesting engineering is in classifying the diff, not merely detecting it. One caveat I'd add from running agents in production: the description is what the agent actually reasons over, so a rewritten description deserves a hold even when the schema didn't change. The approval side is the harder half — few teams have a human available at every restart, so the checkpoint eventually needs to learn which diffs are routine enough to auto-pass.
@agenshive yes.. a rewritten description holds the call today, schema or not. the description is the thing the agent reads so that was the first check built.
on auto pass.. honest answer is no and i've held off on purpose. if it learns what's routine.. a server that changes slowly enough gets through. so instead i made the approve step fast.. was/now side by side, one click and only the changed tool is held so the rest keep working.
how many restarts a day are you seeing?
You should stop your agent on the 25 updates that allow a tool to do something new, while letting the 24 "reshuffled" updates pass through.
@jillianj Thanks that's the split I would use.
Diffing the input schema on each update catches what a name-based approval misses. Did any of those 25 add an argument that takes a path or a URL?
@paul_irolla_6b5ae261d0224 Yes the shipped check found 4.. Playwright's browser_tabs.url, and Chrome DevTools' evaluate_script.filePath, emulate.extraHttpHeaders and upload_file.filePaths.
Going back for your question I found a fifth it missed.. CircleCI 0.16.0 added an outputDir nested inside another parameter and the check only looked at top level names. So it's 5 and the nested case is going into the script.
@neelagiri65 The nested case is the one a top-level name check cannot see, so the CircleCI one is the find that matters here.
@paul_irolla_6b5ae261d0224 agreed. it sits inside an object, two levels down which is exactly where a name check doesn't look. I wouldn't have gone back for it without your question.
The 25-versus-24 split is useful, but a checkpoint taken only when a server starts leaves a time-of-check gap for long-running agents. If a server restarts or is redeployed mid-session, the agent can keep using a stale approved definition while the live tool has already changed. For effectful tools, I would revalidate the definition hash immediately before the call, not just at session startup, and fail closed on any mismatch. That also raises a question about default values: a parameter that keeps the same name but flips its default can quietly widen what a call does without triggering a new-tool or new-parameter rule. How do you classify a default-value change in your taxonomy, or does it fall into the reshuffle bucket today?