There's a good checklist going around dev.to for vetting an MCP server before you wire it into an agent. Four things to look at. Tool surface area:...
For further actions, you may consider blocking this person and/or reporting abuse
Payments make auth part of every eval, not a separate row. The tool call, user intent, merchant context, amount, idempotency key, and approval boundary all need to line up before the action should be considered safe.
Right, auth stops being a row and becomes the join condition. Tool call, intent, merchant, amount, idempotency key, approval boundary all have to line up in the same breath, and if any one of them is fuzzy the action isn't safe yet. The line I keep drawing is which of those the server can actually assert versus which the caller has to carry itself. Most servers today can only speak to a couple of them, so the rest lands on you whether you planned for it or not.
That server/caller split is the part I keep coming back to. The server can assert resource ownership, idempotency state, and maybe merchant scope; the caller has to carry user intent, approval freshness, and why this amount is legitimate now. If those are not joined into one signed or at least auditable decision, auth looks green while the business action is still unsafe.
That split is the thing I'd want written down before anyone wires an agent to a payment rail. Server side: ownership, idempotency state, maybe merchant scope. Caller side: user intent, how fresh the approval is, why this amount is legitimate right now.
The failure you're naming is the quiet one, where every individual check passes and the action is still wrong, because nothing joined them into one decision anyone can reconstruct afterwards. Regulated payments forces you to produce that joined record whether you want it or not. Most agent stacks have nowhere to put it.
That server-versus-caller split is the uncomfortable part. If the server can assert only merchant and amount, but approval boundary and idempotency context live in the caller, the eval has to test the whole chain or it is testing a fantasy version of auth.
"Testing a fantasy version of auth" is the phrase for it. An eval that scores the server in isolation is measuring a component that cannot make the guarantee on its own, then reporting a pass.
What I'd want instead is the eval running the whole path: mint the intent, call the tool, drop the connection, call again, then check that the money moved exactly once. That's a chain test, and it fails for reasons no single-row checklist would have predicted.
The version we landed on is blunter than that. The eval has to be able to reconstruct, after the fact, why this specific amount moved to this specific party at this specific moment. If the trace can't answer that, the chain wasn't tested. The parts were.
This is the right altitude for payments. The idempotency key has to belong to the user's intent, not the model's current turn, because retries are exactly where the agent stops being the interesting part. I also like the read/write split. A server can be safe for balance checks and still too blunt for money movement unless the mandate is narrow enough to audit later.
Exactly — "the model's current turn" is the phrase I was reaching for and didn't quite land in the post. That's the seam. And your second point is the one people skip past: a server can pass a balance-check eval clean and still be the wrong thing for money movement, because the eval never asked the mandate to be narrow enough to reconstruct later. Same server, two different risk classes.
Really useful breakdown of mandate vs key. The time-boxed and revocable properties are what actually matter operationally when something goes sideways and you need to kill a credential without rotating everything.
On your closing question: I am seeing some movement here. Teams building agent-native payment infrastructure (CAI Labs being one example) are designing settlement that lives inline where the agent executes, so the server can express retry safety guarantees rather than punting it to whichever client happens to call the tool.
Pushing retry-safety down so the server expresses the guarantee instead of every caller reinventing it is the right direction. The bar I'd hold any of these approaches to is verifiable, not just declared. If the server says it's idempotent, I still have to trust it; if I can prove a replay collapsed to a single charge, I don't have to. That's the difference between a safety property and a marketing line, and it's the thing I'd want to see before letting settlement live inline with execution. How does that assurance actually surface to whoever's on the other end of the tool call?
On your closing question: the spec does have annotations for exactly this — readOnlyHint, destructiveHint, idempotentHint — but "hint" is doing real work there. They're advisory, declared by the server about itself, and nothing verifies them. So in practice it's still on the caller, and your NON_RETRYABLE set is the honest version of it: a list the client owns rather than a claim the server makes.
The part I'd add is what to do when you don't control where the key is minted.
Your example works because the client creates the intent before the tool is ever called. Often you're on the receiving end of somebody else's retry instead — a webhook, an inbound email, any queue with at-least-once delivery — and no key ever arrives.
There you have to derive one from the payload: hash the fields that make it the same real-world event, and treat a match inside a short window as a redelivery. It works because a redelivery is byte-identical by definition, while a genuine second event almost never is. Cruder than a real key, and you have to choose the fields and the window deliberately rather than by feel — but it turns "have I already done this" into something answerable without the sender's cooperation.
Same shape as your payment case, either way: the guarantee has to sit below the layer that's allowed to be jittery.
This is the comment I was hoping someone would leave. You're right that the hints exist, and "hint" is the whole problem. readOnlyHint, destructiveHint, idempotentHint are the server describing itself, unverified, so I can't build a boundary on them any more than I'd trust a query that swears it's a SELECT. The NON_RETRYABLE set is ugly precisely because the client owns it.
The derived-key point is the part I left out and shouldn't have. At-least-once delivery is exactly where no key arrives and you're reconstructing intent from the payload. Hashing the fields that make it the same real-world event, matched inside a short window, is the same fallback we use on inbound webhooks, and choosing the fields and the window is the whole game. Too many fields and a legitimate retry looks new; too few and two real events collide. I've never found a window that generalises, so we set it per event type by hand and lean on the natural cadence of that event. Have you landed on a saner default, or is it always hand-tuned for you too?
Hand-tuned here too, and I've stopped hunting for the general one — but I think the reason why is the useful bit.
The window is only load-bearing when the key is fuzzy. Where I can match on something byte-identical, the window stops being the thing making the decision and becomes a safety net, so it can be generous and it's only ever wrong in the safe direction. Two genuinely different events that serialise to the same bytes, close together, basically don't happen.
The moment I have to normalise or fuzzy-match to build the key, the window is suddenly doing the actual work, and that's where I've never found a default either. So I've come round to treating "do I need a window at all" as the first question rather than "how long should it be".
On picking the fields, the rule that's held up: include what makes it the same event from the sender's point of view, then explicitly drop anything they regenerate per attempt — timestamps, message ids, attempt counters, retry headers. When a legitimate retry looks new to me it's almost never that I used too many fields, it's that one volatile field crept in that had no business being in the key.
"Do I need a window at all" as the first question is the reframe I was missing. You're right that the window only carries weight when the key is fuzzy, and I'd been treating window length as the tuning knob when the real decision happens one step earlier.
The volatile-field point matches what we see. Every false-new we've had traced back to one field that regenerates per attempt sneaking into the hash. Usually a timestamp somebody added for debugging and never took out.
The one I still don't have a clean answer for is fields that are stable per attempt but not byte-identical across senders. Same bank, same payment, two slightly different reference string formats depending on which endpoint fired. Normalising it makes the key fuzzy again and puts me right back in window territory.