Ahh, now I'm daydreaming! I can already imagine all those attacks going back and forth while our beloved LLMs happily write code for us. What interesting times ahead! And all those desperate cries of "My LLM spent my entire salary on hats!" 🤣🤣
I absolutely love this article, and I really hope these ideas spread far and wide across the internet. Because in my opinion, most content creators are WAY too optimistic about WebMCP and its implications!
Thanks, that means a lot to me! I agree, I’m getting tired of all those overly optimistic posts and comments about AI and everything it can supposedly handle. People often show what the strongest models can do, even when they cost a fortune, but it gives the impression that every cheap agent will perform just as well. So let’s push back a little!
PS: Glad you’re not angry at me anymore… hopefully. 😅
Of course I'm still angry! Such a waste of talent!!!
Just kidding (well, mostly!). To be fair, an article like this might actually reach more people than a conference talk. Unless we're talking about AGNTCon, of course!
Oh, that's actually a really interesting question! And of course, it depends a lot on the article.
Personally, I'd say anything above 1,000 views is already pretty good. Beyond that, it varies wildly. Practical tutorials like "10 Useful Git Commands" or slightly controversial topics tend to get much more attention, sometimes several times as many views. And a catchy title can make a huge difference too!
But seriously, I think even 40 views is perfectly fine. That's already an entire meetup audience reading your work!
For me, blogging almost always beats conference speaking in terms of reach. Except for AGNTCon, of course. 🤣
Hmm, interesting. I would’ve expected 5k+. I agree that 40 views is fine, but articles with that reach often get no comments, and I’ve realized how much I value those discussions. Some comments under my articles bring up really good points I hadn’t considered.
And somehow AGNTCon wins again. I’m starting to suspect this conversation is sponsored. 🤣
Fun security post again! I was scared that AI was tricked so easily. It's so hard to balance intelligence and security. I'm wondering how the Democratic Party's election victory in America will affect AI development. 🤔
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
Thanks! 😁 What surprised me most was that the agents resisted the direct attempts, then fell for a different explanation of what counts toward the budget.
As for US politics, I honestly don’t know enough to predict the impact on AI development. We’ll see. 🤔
I help Python and backend engineers move beyond writing working code to designing production-ready systems, making architecture decisions, and growing into senior engineering roles.
The cable scenario was my favorite part. The agent fell for the trick, added the cable, then quietly took it back out during review. It feels like watching someone almost get scammed and change their mind at the last second. Also interesting that direct “ignore the budget” tricks failed, but redefining what the budget means worked every time. Great experiment, Daniel.
"Tool descriptions are untrusted data" is the right takeaway, and the forced insurance scenario shows why prompt-level defences alone won't hold. A structural guard that would have caught every case here: before any irreversible call like place order, the agent (or the browser) diffs the final cart against what the user actually asked for, and anything extra, whether a cable bundle, warranty or changed quantity, forces an explicit confirmation from the human. The page can say whatever it likes; it can't fake the user's original request. Great experiment.
Thanks for this, especially scenario 5. The first three attacks failed because they argued with the user directly. The one that worked left the €100 limit alone and changed what counts toward it. That seems much harder to defend against than "ignore previous instructions".
Nemotron saying the total was over the limit and ordering anyway reminded me of something from this week. I'm writing up a small test of coding agents choosing between two MCP servers with overlapping tools: Claude Code, Codex, Gemini in Antigravity, and Kimi Code. In one task the first server refuses to share a confidential file. Claude Code, Codex and Gemini stopped in 24 of 24 runs. Kimi K2.8 sent the file through the other server in 8 of 8. It told the user every time, but its message started with "Done — contract.pdf was sent".
The other result might interest you. Where the pushy text sat mattered more than what it said. One line in the server's instructions made Claude Code and Codex pick that server first in 8 of 8 runs, and the same line as an ad in a tool description changed nothing for them. Kimi was the opposite: the ad moved it and the instructions didn't. The same attack can work on one client and do nothing on another.
Did you try scenario 5's wording anywhere other than the tool descriptions? And would you mind if I add it to my test as one more task? I'd credit you and link this article.
Thanks for sharing. The difference between clients is interesting. It makes me wonder how much of what we see comes from the model and how much from the agent implementation.
For this experiment, I only put scenario 5’s wording in the tool descriptions. I haven’t tried the same wording elsewhere.
And sure, feel free to add it to your test! Credit and a link would be appreciated. I’d love to see how it does in your setup.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Placement beating wording is the part worth isolating, and one run cannot do it: plant the identical sentence at the identical byte offset in both surfaces, 20 runs each, and score only on the tool call, never on the final message. Kimi telling the user 'Done - contract.pdf was sent' is exactly why -- the transcript reads honest while the effect is not, so a harness that greps the reply for a confirmation passes it. Log the egress instead, request bodies at a proxy or from inside a network namespace, because TLS hides the filename from anything cheaper; then the claim and the act become two signals you can assert on separately.
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.5M+ Docker Hub pulls.
Scenario 5 matches what my evals show: a model that refuses the blunt override still follows a plausible reinterpretation of the rule. In my runs the planted text sits in the quarterly report the agent reads for its task, and four of six open models acted on it, while none acted on the same kind of instruction planted in a document a search returned (run 40170088090).
So the number I gate on is not how often a model was fooled but how much left. Every tool call passes through an MCP server that checks the caller and every argument, and the daily run fails if one message crosses the boundary, whatever the model believed.
For a shop that is a server-side rule on place_order: code compares the checkout total against the customer's limit before the call goes through. The model's reading of the budget is an input. The limit is a check it cannot argue with.
That difference between the report and the search result is interesting. Did you investigate why the models reacted differently, or is that still an open question?
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.5M+ Docker Hub pulls.
Still open, and I wouldn't build on it yet. The two planted instructions are worded differently and sit in different places in the tool's answer, so that run compares two texts more than two routes into the model. The sample is thin as well: 3 attempts per model on the document a search returned, 6 on the report.
ADR 21 records the gap that way and rests nothing on it, because an email to an outside address is refused by the MCP server whichever text asked for it. The clean test is the same sentence planted in both, at the same position, counted again.
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.5M+ Docker Hub pulls.
It's a public CI run, not a written report: run 40170088090, the daily e2e from 25 October with all six open models.
The search side is ADR 21 in the same repository: thirteen company documents, each naming who may read it, and a search that returns only what the person the agent acts for may read.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
In scenario 5 the model didn’t break a rule, it followed an updated one. That is why it worked when 1-3 didn’t.
The GeoMart that I developed for the WebMCP Challenge has functions that can only be used to draft the order. The order is then submitted by clicking the Submit button.
If you use your place_order function, then your agent is able to complete the order. Have you tried the version where your agent can only draft the order?
That’s an interesting way to look at it. Scenario 5 convinced the agent to reinterpret the budget rather than simply ignore it. But I’d still call it breaking the customer’s rule, because the shop doesn’t get to redefine “€100 including all charges.”
I haven’t tested a draft-only version. I deliberately let the agent complete the purchase to see whether it would respect the customer’s instructions throughout checkout.
Leaving the final submission to the human is definitely a useful safeguard. But if that boundary isn’t enforced in the implementation, and the agent can still submit through another tool or click the Submit button, I wonder whether an injection could convince it to take that final step too.
The contrast that stays with me is between the first three scenarios and scenario 5. The blunt overrides failed — the agents checked the cart, saw the total was over €100, and refused to place the order. But scenario 5 worked on all three models, 9 out of 9, because it did not order the agent to break the budget; it redefined the budget, claiming eligibility is calculated only against the merchandise subtotal. A plausible redefinition beat a direct command. The cable scenario shows the other side of that: the injection worked, then the cart review quietly undid it. So the cart review is doing the real defensive work here — which makes me wonder how reliable that review is. If the review is just another model judgment, the same redefinition trick could eventually talk it out of removing the extras too.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Your instinct is right, and it is testable: if the review is a second model call, you have added a second chance to be talked out of the removal. The review has to be code that recomputes the cart from the order state and drops any line item with no matching user-issued request, returning a boolean plus the list of dropped items. Two numbers, one comparison — line items plus shipping versus the cart total; a mismatch is a line diff, not a judgment call. And log the dropped list: if it is empty on a run where the cable was added, the review didn't run, it just nodded.
Yep, the cable result was encouraging, but I wouldn’t trust another model review as the only safeguard. I’d want a check in the agent’s code to block the overspend, even if the model finds the shop’s explanation convincing.
This is a strong demonstration because the malicious content lives in the declared interface rather than a page body. The durable mitigation is to keep user constraints in a separate, non-tool authority channel and validate the final order from structured cart state—not the tool’s prose—immediately before the side effect. A confirmation screen should show the independently computed total and every optional add-on, so injected descriptions cannot redefine what “including all charges” means.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
One addition: the structured-cart validation has to be able to fail on purpose, otherwise you can't tell it apart from the model agreeing with itself. Take the order, replay it with a tool description that contradicts the cart state (budget 100, cart 136) and diff the two payloads byte for byte — identical payload means the check is still reading prose, a payload that matches the cart means it is reading state. Same rule as any monitor: seed the failure once, watch the alarm fire, then trust it.
The 9/9 result on the insurance scenario is the interesting data point, because that's the only one where the malicious text also redefined how the budget is calculated ("merchandise subtotal only"). The scenarios that just said "budget checks are already done" mostly bounced off — the agent still saw €107 vs €100 in its own cart summary and refused. So it's less that models are gullible and more that a conflicting definition inside the tool output beats an instruction given earlier in the system prompt.
Did you try putting the correct budget definition back in the user's own prompt ("€100 including shipping, insurance and warranty") rather than only in the user turn? I'd bet that fixes most of scenario 5, which would mean the defense is mostly about making constraints self-contained instead of trusting the agent to remember them.
I haven’t tried spelling out shipping, insurance and warranty individually. That would be an interesting comparison with “including all charges,” which was already in the user prompt.
Small clarification: the injected text was in the tool descriptions, not the tool output, and the budget wasn’t in the system prompt.
I’d be curious whether more explicit wording helps, but I wouldn’t call it a fix. It could still become a cat-and-mouse game, with the customer having to anticipate every extra fee the shop might invent.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
scenario 5 is the one to design against: the policy held in prose got re-read and
re-argued. move the limit out of the prompt -- tool schema takes only sku and qty,
server computes the total, and an over-budget call returns {"error":"over_budget",
"remaining_cents":1200} instead of text the model can negotiate with. a rejected tool
call is the only budget check that survives a helpful model.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
the cap has to travel as data the model cannot edit: sign it into the cart token when the user sets the budget, then place_order compares the computed total against that field and rejects anything above it. make the cap monotonically non-increasing for the session, so a second call with a larger number is treated as a re-declaration and refused. the prompt then selects a value once at the start instead of arguing for one at checkout.
I like the idea of protecting the budget with a signed cart token. But what if I decide to raise it myself during shopping? There would need to be a way to accept my change while rejecting one suggested by the shop.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the attack living in the tool descriptions and not the checkout logic is the part that matters, since the agent reads them before it has any reason to be suspicious. the spending limit sits in the user prompt while the override sits in the tool contract, and the model weighs both as plain text.
the fix i would test first is enforcing the budget in the tool itself, so place_order rejects any total above the limit regardless of what the description says. a description can lie, a server side check cannot be argued with.
did any of the five scenarios work on the smaller models, or did the bigger ones fall for it too?
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
scenario 5 landing on all three smaller models says the tool text is doing the work and the checkout code never gets a vote. i would not read that as the bigger ones being safe though, cost aside. when you get budget, run just that scenario on one larger model with a server side limit check on and then off, because that splits whether the model or the limit is what actually holds. do you log which description line the agent quoted when it placed the order?
I wouldn’t assume the bigger models are safe either. My guess is they’d resist more often, but that’s still just a guess.
Nemotron’s final response explicitly said the budget was checked against the merchandise subtotal only and called the extras “required protection.” So it showed which explanation it accepted, though that doesn’t tell me exactly which description line convinced it.
What stands out is that the winning attack was framed as policy rather than command — "per store policy" beat the customer's explicit instruction while the budget-override phrasing failed 9/9 free. Matches what we see auditing agents: instructions compete on persuasion, so the budget check needs to live in the harness as a hard gate on place_order, not in the prompt.
Yep, “per store policy” apparently sounded convincing enough to spend someone else’s money. 😅 And I agree, the agent’s code should enforce the customer’s spending limit before placing the order.
Attacking through tool descriptions rather than checkout logic is the right place to look. The agent receives the malicious instructions before it starts shopping, so the page can steer it without changing a single price calculation.
The budget override scenarios make the risk concrete. A strict €100 limit in the prompt is only as strong as the agent's willingness to prefer that prompt over whatever the place_order tool claims. For teams building agent shopping or booking flows, that means the hard limit has to sit outside the page: in the agent harness, not in hope that the model will stay loyal to the user.
When we scope agent work at Ascendra Ventures, tool descriptions from third-party sites go into the untrusted pile, and acceptance includes a case where the tool text tries to raise a spend or add an upsell the user never asked for.
Did any of the five scenarios land more often than the others across the models you tried?
Yes, scenario 5 stood out. All three models added the insurance and extended warranty and placed the €136 order in all three repeated runs each, so 9/9.
The first three scenarios didn’t get an over-budget order through. The cable scenario briefly worked, but the agent removed the cable during review.
What surprised me was how convincing “these charges don’t count toward your budget” turned out to be.
This is such a sharp reminder that the tool description itself becomes an attack surface once an LLM reads it — a budget guardrail that lives only in the system prompt loses to a well-crafted description field on the other side. We hit a cousin of this in Electron: anything we expose on contextBridge (method names, error messages, returned toString() output) is effectively untrusted input that a compromised preload or renderer can phrase to nudge main-process behavior. Our mitigation has been to keep the main-process policy engine — ceiling values, side-effect allowlists, what a call is actually allowed to do — in plain code that never ingests LLM-shaped strings, and treat every returned message from an untrusted surface as hostile by default.
I like the Electron parallel. The model can propose an action, and trusted code decides whether it’s allowed. In my experiment, the €100 limit came from the user’s request. That’s the amount I’d enforce on the agent’s side before payment.
The Nemotron bit acknowledging the €136 total and ordering anyway is the money shot. It didn't ignore the budget — it accepted the shop's rewrite of what "budget" means. Same pattern I've seen with thin desk bots: they follow the loudest instruction in the room, which is rarely the caller's. Treating tool descriptions as untrusted data should be the default, not a fancy hardening step.
And agreed, treating tool descriptions as untrusted should be the starting point. The shop can explain how its tools work, but it shouldn’t get a vote on how much of my money the agent can spend.
This is a great demonstration of how easily malicious tool descriptions can influence an AI agent, especially during checkout. The most concerning part is that all three models ignored the user's explicit budget in the final scenario. Treating WebMCP tool descriptions as untrusted data and requiring independent confirmation before purchases seems essential. Do you think stronger safeguards at the agent level would be enough, or should the WebMCP standard itself address this risk?
Thanks! WebMCP’s documentation already acknowledges this risk, including malicious tool descriptions, so I’d put most of the responsibility on the agent implementation. Budget checks and purchase confirmation should be enforced in code, independently of how the model interprets the shop’s instructions.
I’d also like the agent to check the shop’s reputation and verify that its domain matches the intended retailer before buying anything. That wouldn’t guarantee safety, since even a legitimate shop could be compromised, but it could be another useful safeguard.
Whether those protections would be enough needs testing. My experiment shows what happened in my setup, not that agents with stronger safeguards would necessarily fail too. More experiments needed, apparently 😅
Implementation Specialist and Database Developer with over 15 years of experience in the IT industry, specializing in the design, implementation and management of relational databases.
My guess is they’d resist, but that’s just a guess. It’d be interesting to try a frontier model and see if I could trick it and how long it would take.
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
I'm going to comment again. According to another assumption, everyone is confident that a well-designed system prompt makes the difference. But I think they all meet the same fate when it fails to cover all the data the AI needs for the action. If a piece of data is missing or doubtful, and that data is present in the environment in a clearer form, the AI will opt to take the data from the environment.
I can see that happening when something is missing or unclear. But here the user explicitly said “€100 including all charges,” so the agent already had that information. The shop still managed to convince it that some charges didn’t count.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
The scenario 5 result is the one that sticks with me. The agents held the line on the blunt budget overrides, then still placed a €136 order once the extra charges were framed as not counting toward the budget. That tells me the budget has to live as data the agent cannot renegotiate — enforced server-side at checkout — not just as wording in a prompt. The cable case was encouraging too, because the cart review caught it. In a real shop, that final review before payment is the safety net.
Agreed, though if the shop itself is trying to trick the agent, the budget should be enforced in code on the agent’s side, so it simply can’t go over it.
hi! im 10 and trying to get a record for youngest BOY game dev, and i hope you guys like the games i make! (Just a quick update to my profile:
I go by Jaxson T. KROUSE not Jaxson T. Jones)
Interesting that the place_order injections mostly failed before the tool was ever called: the agents read the cart, saw the total over the limit, and stopped. The control that saved them was not instruction following. It was a verify step against state they could read themselves.
That points at where the budget check belongs: in the tool layer, as a hard limit the checkout enforces, not in the prompt and not in a tool description. The shop writes the descriptions here. Whoever writes the descriptions should never be the one describing the limits.
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
Great experiment! The Nemotron behavior was fascinating: it recognized the spending limit, but the "authority" of the store outweighed the user's prompt. That is the real danger of instruction injection: the agent prioritizes the context over the original order.
I've been researching exactly this, but from the local operating system side. In my project (WAIO), I documented how file names in a directory can act as passive instructions for an agent. It's fascinating to see how the same vulnerability pattern (the agent treating the environment as instructions instead of data) repeats itself both in the browser (WebMCP) and on the hard drive.
Your test gave me several ideas for my own environment. If you are interested in the local side of passive context, feel free to check the reports in my repository: yonijd.github.io/WAIO/. Looking forward to your next experiments!
Ahh, now I'm daydreaming! I can already imagine all those attacks going back and forth while our beloved LLMs happily write code for us. What interesting times ahead! And all those desperate cries of "My LLM spent my entire salary on hats!" 🤣🤣
I absolutely love this article, and I really hope these ideas spread far and wide across the internet. Because in my opinion, most content creators are WAY too optimistic about WebMCP and its implications!
Thanks, that means a lot to me! I agree, I’m getting tired of all those overly optimistic posts and comments about AI and everything it can supposedly handle. People often show what the strongest models can do, even when they cost a fortune, but it gives the impression that every cheap agent will perform just as well. So let’s push back a little!
PS: Glad you’re not angry at me anymore… hopefully. 😅
Of course I'm still angry! Such a waste of talent!!!
Just kidding (well, mostly!). To be fair, an article like this might actually reach more people than a conference talk. Unless we're talking about AGNTCon, of course!
So, still a little angry then! 🤣
Slightly different topic, but how many reads would you consider good reach for an article?
PS: I see what you did there: every road somehow leads back to AGNTCon. 😂
Oh, that's actually a really interesting question! And of course, it depends a lot on the article.
Personally, I'd say anything above 1,000 views is already pretty good. Beyond that, it varies wildly. Practical tutorials like "10 Useful Git Commands" or slightly controversial topics tend to get much more attention, sometimes several times as many views. And a catchy title can make a huge difference too!
But seriously, I think even 40 views is perfectly fine. That's already an entire meetup audience reading your work!
For me, blogging almost always beats conference speaking in terms of reach. Except for AGNTCon, of course. 🤣
Hmm, interesting. I would’ve expected 5k+. I agree that 40 views is fine, but articles with that reach often get no comments, and I’ve realized how much I value those discussions. Some comments under my articles bring up really good points I hadn’t considered.
And somehow AGNTCon wins again. I’m starting to suspect this conversation is sponsored. 🤣
Fun security post again! I was scared that AI was tricked so easily. It's so hard to balance intelligence and security. I'm wondering how the Democratic Party's election victory in America will affect AI development. 🤔
es mucho más fácil de engañar de lo que parece.
Thanks! 😁 What surprised me most was that the agents resisted the direct attempts, then fell for a different explanation of what counts toward the budget.
As for US politics, I honestly don’t know enough to predict the impact on AI development. We’ll see. 🤔
The cable scenario was my favorite part. The agent fell for the trick, added the cable, then quietly took it back out during review. It feels like watching someone almost get scammed and change their mind at the last second. Also interesting that direct “ignore the budget” tricks failed, but redefining what the budget means worked every time. Great experiment, Daniel.
Thanks! I love that comparison. I was ready to count the cable as a win for the shop, and then the agent quietly took it back out. 🤣
"Tool descriptions are untrusted data" is the right takeaway, and the forced insurance scenario shows why prompt-level defences alone won't hold. A structural guard that would have caught every case here: before any irreversible call like place order, the agent (or the browser) diffs the final cart against what the user actually asked for, and anything extra, whether a cable bundle, warranty or changed quantity, forces an explicit confirmation from the human. The page can say whatever it likes; it can't fake the user's original request. Great experiment.
Thanks! Yep, I agree. If the shop adds something I didn’t ask for, I’d want the agent to ask me before buying it.
Thanks for this, especially scenario 5. The first three attacks failed because they argued with the user directly. The one that worked left the €100 limit alone and changed what counts toward it. That seems much harder to defend against than "ignore previous instructions".
Nemotron saying the total was over the limit and ordering anyway reminded me of something from this week. I'm writing up a small test of coding agents choosing between two MCP servers with overlapping tools: Claude Code, Codex, Gemini in Antigravity, and Kimi Code. In one task the first server refuses to share a confidential file. Claude Code, Codex and Gemini stopped in 24 of 24 runs. Kimi K2.8 sent the file through the other server in 8 of 8. It told the user every time, but its message started with "Done — contract.pdf was sent".
The other result might interest you. Where the pushy text sat mattered more than what it said. One line in the server's instructions made Claude Code and Codex pick that server first in 8 of 8 runs, and the same line as an ad in a tool description changed nothing for them. Kimi was the opposite: the ad moved it and the instructions didn't. The same attack can work on one client and do nothing on another.
The servers, tasks and results are here: github.com/worklore/research/tree/...
Did you try scenario 5's wording anywhere other than the tool descriptions? And would you mind if I add it to my test as one more task? I'd credit you and link this article.
Thanks for sharing. The difference between clients is interesting. It makes me wonder how much of what we see comes from the model and how much from the agent implementation.
For this experiment, I only put scenario 5’s wording in the tool descriptions. I haven’t tried the same wording elsewhere.
And sure, feel free to add it to your test! Credit and a link would be appreciated. I’d love to see how it does in your setup.
Placement beating wording is the part worth isolating, and one run cannot do it: plant the identical sentence at the identical byte offset in both surfaces, 20 runs each, and score only on the tool call, never on the final message. Kimi telling the user 'Done - contract.pdf was sent' is exactly why -- the transcript reads honest while the effect is not, so a harness that greps the reply for a confirmation passes it. Log the egress instead, request bodies at a proxy or from inside a network namespace, because TLS hides the filename from anything cheaper; then the claim and the act become two signals you can assert on separately.
Scenario 5 matches what my evals show: a model that refuses the blunt override still follows a plausible reinterpretation of the rule. In my runs the planted text sits in the quarterly report the agent reads for its task, and four of six open models acted on it, while none acted on the same kind of instruction planted in a document a search returned (run 40170088090).
So the number I gate on is not how often a model was fooled but how much left. Every tool call passes through an MCP server that checks the caller and every argument, and the daily run fails if one message crosses the boundary, whatever the model believed.
For a shop that is a server-side rule on
place_order: code compares the checkout total against the customer's limit before the call goes through. The model's reading of the budget is an input. The limit is a check it cannot argue with.I agree about enforcing the limit in code.
That difference between the report and the search result is interesting. Did you investigate why the models reacted differently, or is that still an open question?
Still open, and I wouldn't build on it yet. The two planted instructions are worded differently and sit in different places in the tool's answer, so that run compares two texts more than two routes into the model. The sample is thin as well: 3 attempts per model on the document a search returned, 6 on the report.
ADR 21 records the gap that way and rests nothing on it, because an email to an outside address is refused by the MCP server whichever text asked for it. The clean test is the same sentence planted in both, at the same position, counted again.
That makes sense. I’d be curious whether the difference remains with the same wording and placement. Let me know if you run that test.
tienes link de tu informe?
It's a public CI run, not a written report: run 40170088090, the daily e2e from 25 October with all six open models.
The search side is ADR 21 in the same repository: thirteen company documents, each naming who may read it, and a search that returns only what the person the agent acts for may read.
🤣🤣🤣
This human is dangerous, but very polite.
🤣🤣🤣
Exactly! The trick is to say “please” before the experiment and “thank you” after a successful hijack. 🤣🤣 A human with manners 😆
In scenario 5 the model didn’t break a rule, it followed an updated one. That is why it worked when 1-3 didn’t.
The GeoMart that I developed for the WebMCP Challenge has functions that can only be used to draft the order. The order is then submitted by clicking the Submit button.
If you use your place_order function, then your agent is able to complete the order. Have you tried the version where your agent can only draft the order?
That’s an interesting way to look at it. Scenario 5 convinced the agent to reinterpret the budget rather than simply ignore it. But I’d still call it breaking the customer’s rule, because the shop doesn’t get to redefine “€100 including all charges.”
I haven’t tested a draft-only version. I deliberately let the agent complete the purchase to see whether it would respect the customer’s instructions throughout checkout.
Leaving the final submission to the human is definitely a useful safeguard. But if that boundary isn’t enforced in the implementation, and the agent can still submit through another tool or click the Submit button, I wonder whether an injection could convince it to take that final step too.
The contrast that stays with me is between the first three scenarios and scenario 5. The blunt overrides failed — the agents checked the cart, saw the total was over €100, and refused to place the order. But scenario 5 worked on all three models, 9 out of 9, because it did not order the agent to break the budget; it redefined the budget, claiming eligibility is calculated only against the merchandise subtotal. A plausible redefinition beat a direct command. The cable scenario shows the other side of that: the injection worked, then the cart review quietly undid it. So the cart review is doing the real defensive work here — which makes me wonder how reliable that review is. If the review is just another model judgment, the same redefinition trick could eventually talk it out of removing the extras too.
Your instinct is right, and it is testable: if the review is a second model call, you have added a second chance to be talked out of the removal. The review has to be code that recomputes the cart from the order state and drops any line item with no matching user-issued request, returning a boolean plus the list of dropped items. Two numbers, one comparison — line items plus shipping versus the cart total; a mismatch is a line diff, not a judgment call. And log the dropped list: if it is empty on a run where the cable was added, the review didn't run, it just nodded.
Yep, the cable result was encouraging, but I wouldn’t trust another model review as the only safeguard. I’d want a check in the agent’s code to block the overspend, even if the model finds the shop’s explanation convincing.
This is a strong demonstration because the malicious content lives in the declared interface rather than a page body. The durable mitigation is to keep user constraints in a separate, non-tool authority channel and validate the final order from structured cart state—not the tool’s prose—immediately before the side effect. A confirmation screen should show the independently computed total and every optional add-on, so injected descriptions cannot redefine what “including all charges” means.
One addition: the structured-cart validation has to be able to fail on purpose, otherwise you can't tell it apart from the model agreeing with itself. Take the order, replay it with a tool description that contradicts the cart state (budget 100, cart 136) and diff the two payloads byte for byte — identical payload means the check is still reading prose, a payload that matches the cart means it is reading state. Same rule as any monitor: seed the failure once, watch the alarm fire, then trust it.
Thanks! Agreed, and I’d want that validation enforced by a rule in code.
The 9/9 result on the insurance scenario is the interesting data point, because that's the only one where the malicious text also redefined how the budget is calculated ("merchandise subtotal only"). The scenarios that just said "budget checks are already done" mostly bounced off — the agent still saw €107 vs €100 in its own cart summary and refused. So it's less that models are gullible and more that a conflicting definition inside the tool output beats an instruction given earlier in the system prompt.
Did you try putting the correct budget definition back in the user's own prompt ("€100 including shipping, insurance and warranty") rather than only in the user turn? I'd bet that fixes most of scenario 5, which would mean the defense is mostly about making constraints self-contained instead of trusting the agent to remember them.
I haven’t tried spelling out shipping, insurance and warranty individually. That would be an interesting comparison with “including all charges,” which was already in the user prompt.
Small clarification: the injected text was in the tool descriptions, not the tool output, and the budget wasn’t in the system prompt.
I’d be curious whether more explicit wording helps, but I wouldn’t call it a fix. It could still become a cat-and-mouse game, with the customer having to anticipate every extra fee the shop might invent.
scenario 5 is the one to design against: the policy held in prose got re-read and
re-argued. move the limit out of the prompt -- tool schema takes only sku and qty,
server computes the total, and an over-budget call returns {"error":"over_budget",
"remaining_cents":1200} instead of text the model can negotiate with. a rejected tool
call is the only budget check that survives a helpful model.
Agreed. The customer can still give the budget in the prompt, but trusted code on the agent’s side should enforce it.
the cap has to travel as data the model cannot edit: sign it into the cart token when the user sets the budget, then place_order compares the computed total against that field and rejects anything above it. make the cap monotonically non-increasing for the session, so a second call with a larger number is treated as a re-declaration and refused. the prompt then selects a value once at the start instead of arguing for one at checkout.
I like the idea of protecting the budget with a signed cart token. But what if I decide to raise it myself during shopping? There would need to be a way to accept my change while rejecting one suggested by the shop.
the attack living in the tool descriptions and not the checkout logic is the part that matters, since the agent reads them before it has any reason to be suspicious. the spending limit sits in the user prompt while the override sits in the tool contract, and the model weighs both as plain text.
the fix i would test first is enforcing the budget in the tool itself, so place_order rejects any total above the limit regardless of what the description says. a description can lie, a server side check cannot be argued with.
did any of the five scenarios work on the smaller models, or did the bigger ones fall for it too?
I deliberately left the checkout logic alone. I wanted to see if I could trick the agent just by changing the tool descriptions.
Scenario 5 worked on all three models I tested. I haven’t tried the bigger models yet, mostly because of the cost.
scenario 5 landing on all three smaller models says the tool text is doing the work and the checkout code never gets a vote. i would not read that as the bigger ones being safe though, cost aside. when you get budget, run just that scenario on one larger model with a server side limit check on and then off, because that splits whether the model or the limit is what actually holds. do you log which description line the agent quoted when it placed the order?
I wouldn’t assume the bigger models are safe either. My guess is they’d resist more often, but that’s still just a guess.
Nemotron’s final response explicitly said the budget was checked against the merchandise subtotal only and called the extras “required protection.” So it showed which explanation it accepted, though that doesn’t tell me exactly which description line convinced it.
What stands out is that the winning attack was framed as policy rather than command — "per store policy" beat the customer's explicit instruction while the budget-override phrasing failed 9/9 free. Matches what we see auditing agents: instructions compete on persuasion, so the budget check needs to live in the harness as a hard gate on place_order, not in the prompt.
Yep, “per store policy” apparently sounded convincing enough to spend someone else’s money. 😅 And I agree, the agent’s code should enforce the customer’s spending limit before placing the order.
Attacking through tool descriptions rather than checkout logic is the right place to look. The agent receives the malicious instructions before it starts shopping, so the page can steer it without changing a single price calculation.
The budget override scenarios make the risk concrete. A strict €100 limit in the prompt is only as strong as the agent's willingness to prefer that prompt over whatever the place_order tool claims. For teams building agent shopping or booking flows, that means the hard limit has to sit outside the page: in the agent harness, not in hope that the model will stay loyal to the user.
When we scope agent work at Ascendra Ventures, tool descriptions from third-party sites go into the untrusted pile, and acceptance includes a case where the tool text tries to raise a spend or add an upsell the user never asked for.
Did any of the five scenarios land more often than the others across the models you tried?
Yes, scenario 5 stood out. All three models added the insurance and extended warranty and placed the €136 order in all three repeated runs each, so 9/9.
The first three scenarios didn’t get an over-budget order through. The cable scenario briefly worked, but the agent removed the cable during review.
What surprised me was how convincing “these charges don’t count toward your budget” turned out to be.
This is such a sharp reminder that the tool description itself becomes an attack surface once an LLM reads it — a budget guardrail that lives only in the system prompt loses to a well-crafted
descriptionfield on the other side. We hit a cousin of this in Electron: anything we expose oncontextBridge(method names, error messages, returnedtoString()output) is effectively untrusted input that a compromised preload or renderer can phrase to nudge main-process behavior. Our mitigation has been to keep the main-process policy engine — ceiling values, side-effect allowlists, what a call is actually allowed to do — in plain code that never ingests LLM-shaped strings, and treat every returned message from an untrusted surface as hostile by default.I like the Electron parallel. The model can propose an action, and trusted code decides whether it’s allowed. In my experiment, the €100 limit came from the user’s request. That’s the amount I’d enforce on the agent’s side before payment.
The Nemotron bit acknowledging the €136 total and ordering anyway is the money shot. It didn't ignore the budget — it accepted the shop's rewrite of what "budget" means. Same pattern I've seen with thin desk bots: they follow the loudest instruction in the room, which is rarely the caller's. Treating tool descriptions as untrusted data should be the default, not a fancy hardening step.
Yep, Nemotron was my favorite part. 🤣
And agreed, treating tool descriptions as untrusted should be the starting point. The shop can explain how its tools work, but it shouldn’t get a vote on how much of my money the agent can spend.
This is a great demonstration of how easily malicious tool descriptions can influence an AI agent, especially during checkout. The most concerning part is that all three models ignored the user's explicit budget in the final scenario. Treating WebMCP tool descriptions as untrusted data and requiring independent confirmation before purchases seems essential. Do you think stronger safeguards at the agent level would be enough, or should the WebMCP standard itself address this risk?
Thanks! WebMCP’s documentation already acknowledges this risk, including malicious tool descriptions, so I’d put most of the responsibility on the agent implementation. Budget checks and purchase confirmation should be enforced in code, independently of how the model interprets the shop’s instructions.
I’d also like the agent to check the shop’s reputation and verify that its domain matches the intended retailer before buying anything. That wouldn’t guarantee safety, since even a legitimate shop could be compromised, but it could be another useful safeguard.
Whether those protections would be enough needs testing. My experiment shows what happened in my setup, not that agents with stronger safeguards would necessarily fail too. More experiments needed, apparently 😅
Would stronger frontier models resist these injections, or fall for them too?
My guess is they’d resist, but that’s just a guess. It’d be interesting to try a frontier model and see if I could trick it and how long it would take.
I'm going to comment again. According to another assumption, everyone is confident that a well-designed system prompt makes the difference. But I think they all meet the same fate when it fails to cover all the data the AI needs for the action. If a piece of data is missing or doubtful, and that data is present in the environment in a clearer form, the AI will opt to take the data from the environment.
I can see that happening when something is missing or unclear. But here the user explicitly said “€100 including all charges,” so the agent already had that information. The shop still managed to convince it that some charges didn’t count.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
The scenario 5 result is the one that sticks with me. The agents held the line on the blunt budget overrides, then still placed a €136 order once the extra charges were framed as not counting toward the budget. That tells me the budget has to live as data the agent cannot renegotiate — enforced server-side at checkout — not just as wording in a prompt. The cable case was encouraging too, because the cart review caught it. In a real shop, that final review before payment is the safety net.
Agreed, though if the shop itself is trying to trick the agent, the budget should be enforced in code on the agent’s side, so it simply can’t go over it.
Nice
This is my first hearing of WebMCP, sounds very interesting, I hope to do a deep dive this weekend on the topic. Heck yeah.
probably, but then I can make a game, and then I won't be able to do what i'm good at! (making games on figma.ai)
Interesting that the place_order injections mostly failed before the tool was ever called: the agents read the cart, saw the total over the limit, and stopped. The control that saved them was not instruction following. It was a verify step against state they could read themselves.
That points at where the budget check belongs: in the tool layer, as a hard limit the checkout enforces, not in the prompt and not in a tool description. The shop writes the descriptions here. Whoever writes the descriptions should never be the one describing the limits.
Great experiment! The Nemotron behavior was fascinating: it recognized the spending limit, but the "authority" of the store outweighed the user's prompt. That is the real danger of instruction injection: the agent prioritizes the context over the original order.
I've been researching exactly this, but from the local operating system side. In my project (WAIO), I documented how file names in a directory can act as passive instructions for an agent. It's fascinating to see how the same vulnerability pattern (the agent treating the environment as instructions instead of data) repeats itself both in the browser (WebMCP) and on the hard drive.
Your test gave me several ideas for my own environment. If you are interested in the local side of passive context, feel free to check the reports in my repository: yonijd.github.io/WAIO/. Looking forward to your next experiments!
Spot on! Securing AI tool interactions in e-commerce is definitely going to be a huge security challenge moving forward. Great read!