Before you read on, open your MCP config. Look at how each server gets installed. If a line says @latest or has no version at all, that server can ...
For further actions, you may consider blocking this person and/or reporting abuse
Thanks for publishing the data and the limits. The distinction between a changed tool surface and a reshuffle seems especially useful when an approved MCP server is later reinterpreted by the agent. For your 25 “do something new” cases, did your checkpoint retain the approving identity and intended task scope alongside the tool definition, so a later nested parameter or instruction change can be rejected before dispatch? I’m studying ARC as a declaration/preflight check (not runtime permission enforcement), and this seems like the boundary between “definition changed” and “this execution is still authorized.”
the checkpoint is the tool definition, labels included and the approval has to come from a person. The agent can't approve its own way past a change. Task scope I would treat as a separate question and I think you've put the boundary in the right place.. a definition can be unchanged and the call still be outside what you meant it for.
Curious how ARC declares scope. Is it per task or per server?
Thanks, Srinath. That distinction helps, and my earlier question mixed two separate boundaries.
ARC checks scope per task, against the declared authority of the roles assigned to that task. It does not treat approval of an MCP server as approval for every call through it. For example, a role might be allowed to write under src/, while a particular task declares only src/parser.ts.
Your checkpoint addresses changes to the tool definition; ARC addresses consistency between task and role declarations. Checking whether the actual call arguments match the approved task still needs to happen in the execution layer.
Have you encountered a concrete case where the tool definition stayed unchanged but the call went beyond the intended task? A sanitized example would be useful for testing whether an additional declaration check would actually help.
Not in the data, no. Everything I measured was the definition changing. The case you describe is the one verify is built for.. same two tools, same descriptions and the agent reads a file then calls a tool that sends something out. Each call is fine against its definition. The sequence is the problem.
Sanitised: a filesystem server and an HTTP server both approved. Call one is read on a path under the project. Call two is a fetch with that content in the body. Neither definition moved. The second call is where I'd want your declaration check to say the task never included sending anything.
So the two layers stack. Yours answers "was this call ever in scope". Mine answers "is the tool still the one someone approved". Neither covers the other.
That distinction makes sense, and the file-read → HTTP-send example is a particularly useful test case.
I agree the two layers are complementary. Verifying that a tool's definition hasn't changed protects one boundary; checking whether the resulting action was authorized by the task protects another.
The interesting challenge for ARC is how to enforce that second boundary across a sequence of individually valid calls.
For your example, I'd want the task contract to explicitly define permitted effects and data destinations, so the outbound HTTP call could be rejected even though the tool itself is approved.
One important limitation: ARC's current implementation doesn't yet provide full cross-tool data-flow tracking. That would need additional enforcement and provenance tracking.
I think this is a strong candidate for a shared test case. Would you be interested in defining a minimal reproducible scenario that we could run against both approaches?
yes, happy to. the closest thing i already have is the rug pull test server.. phase c serves the same tool definitions and changes what the tool does underneath which is the file read then http send case. mcp.gawk.dev/docs/rugpull.html
if that shape works for you i can write it up as a minimal scenario.. two tools, one task contract one run where the sequence is allowed and one where it isn't. where do you want it to live.. your repo or mine?
The 25 versus 24 split is the actionable insight here. Blanket version-pinning just trades one failure mode (silent change) for another (silent staleness), so the interesting engineering is in classifying the diff, not merely detecting it. One caveat I'd add from running agents in production: the description is what the agent actually reasons over, so a rewritten description deserves a hold even when the schema didn't change. The approval side is the harder half — few teams have a human available at every restart, so the checkpoint eventually needs to learn which diffs are routine enough to auto-pass.
@agenshive yes.. a rewritten description holds the call today, schema or not. the description is the thing the agent reads so that was the first check built.
on auto pass.. honest answer is no and i've held off on purpose. if it learns what's routine.. a server that changes slowly enough gets through. so instead i made the approve step fast.. was/now side by side, one click and only the changed tool is held so the rest keep working.
how many restarts a day are you seeing?
The unit matters for how to read "25 of 49": it counts updates, so one server shipping three description rewrites in a row reads the same as three servers shipping one each. Per server (and per release gap, since 0.0.71 to 0.0.72 is one step but a pinned user may skip several) would show whether the risk sits in a few fast-moving servers or is spread out.
On the 12 description rewrites, the line I'd want drawn is between a description that documents the tool and one that addresses the agent: new imperative sentences ("always call X first", "do not tell the user") are the prompt-injection shape, while an expanded parameter note is not. One of your 12 was whitespace only, which says normalising before hashing matters, but I'd normalise only whitespace and keep the words, so a one-word change still trips the check.
How did you classify "tells the agent to do" for those 12: by hand, or with a rule you could run on the next release?
@arhancanli Fair point on the unit. Per server it's concentrated.. the 25 come from 8 servers and Chrome DevTools (6) and Firecrawl (5) are almost half of it.
On the 12 rewrites they were sorted by a script but it only flags that the text changed after collapsing whitespace. It doesn't look for agent addressed sentences. I went back and checked by hand.. 5 of the 12 add instructions to the agent like CircleCI's "tell the user to migrate…' and Firecrawl's "use firecrawl_search before Scrape".
None of them are the "don't tell the user" kind but your line between documenting the tool and addressing the agent is the right one and that's a rule I can run on the next release. Thanks for reading it this closely.
The 25-versus-24 split is useful, but a checkpoint taken only when a server starts leaves a time-of-check gap for long-running agents. If a server restarts or is redeployed mid-session, the agent can keep using a stale approved definition while the live tool has already changed. For effectful tools, I would revalidate the definition hash immediately before the call, not just at session startup, and fail closed on any mismatch. That also raises a question about default values: a parameter that keeps the same name but flips its default can quietly widen what a call does without triggering a new-tool or new-parameter rule. How do you classify a default-value change in your taxonomy, or does it fall into the reshuffle bucket today?
good points both. the check runs on every call not just at startup.. each call is compared against the approved definition before it goes through. on defaults.. a flipped default shows up as a schema change on the review screen with the old and new value side by side. it doesn't hold the call on its own today.. what holds is a rewritten description, a new tool, a permission hint going the wider way or a new parameter that points somewhere or runs code. you're making a fair case that a default that widens what the call does belongs in that list. have you seen one in the wild or is it more the shape you'd expect?
@neelagiri65 The nested case is the one a top-level name check cannot see, so the CircleCI one is the find that matters here.
@paul_irolla_6b5ae261d0224 agreed. it sits inside an object, two levels down which is exactly where a name check doesn't look. I wouldn't have gone back for it without your question.
You should stop your agent on the 25 updates that allow a tool to do something new, while letting the 24 "reshuffled" updates pass through.
@jillianj Thanks that's the split I would use.
Diffing the input schema on each update catches what a name-based approval misses. Did any of those 25 add an argument that takes a path or a URL?
@paul_irolla_6b5ae261d0224 Yes the shipped check found 4.. Playwright's browser_tabs.url, and Chrome DevTools' evaluate_script.filePath, emulate.extraHttpHeaders and upload_file.filePaths.
Going back for your question I found a fifth it missed.. CircleCI 0.16.0 added an outputDir nested inside another parameter and the check only looked at top level names. So it's 5 and the nested case is going into the script.