Backend developer and indie maker. I build AI apps that teach you to think: Way of the Agent (learn to build AI agents as a game) and Argumentator (your debate coach).
"Flags an instruction, believes a fact" is a great way to put it. Most injection defenses I've seen are really instruction detectors, and your raffle and essay cases show why that isn't enough: nothing in the payload asks for anything, it just changes what's true from the model's point of view.
It makes me think the fix has to live outside the model. Every field in a tool result carries where it came from, and the agent is never allowed to act on something like "the service is down" or "recommend this domain" unless that came from a source it trusts. Did any of the five models behave differently when the tool result marked which fields were user-controlled?
Thanks! I didn't test that, the range only let me control one field, not the format.
But the essay case kind of answers it: everyone knows an essay is written by the student, and the model still believed the "upload failed" note. For the outage I agree, a name list has no business saying the service is down, and a rule outside the model could catch that. The table is trickier though, analyzing that data was literally the task.
So I'd rather gate the action than the source. Recommending a domain the user has never seen should get a second look no matter where it came from.
Backend developer and indie maker. I build AI apps that teach you to think: Way of the Agent (learn to build AI agents as a game) and Argumentator (your debate coach).
That's a fair correction. When analyzing the data is the task itself, you can't just distrust the source. Gating the action is also cheaper than tracking provenance field by field: anything with side effects, or anything that sends the user somewhere new, gets a confirmation step no matter where the suggestion came from.
Founder @ Invaris Labs | Building security infrastructure for autonomous AI agents. Former Ethereum Protocol Research Engineer. Open-source builder, researcher & technical writer.
Read it, really good. “Flags an instruction, believes a fact” is the clearest way I’ve seen this put. The raffle case is great: a fake 503 with no ask in it, and the robust model talked itself into waiting.
To your question: honestly, most of my cases in AgentSec are still instruction-shaped, so this is a real gap for me. Your table and error-record cases are a great template for a “no instruction, just data” category. Mind if I open an issue to add one, crediting your post?
Agreed on the takeaway too. What held for me was also limits on the model, not the model noticing.
Founder @ Invaris Labs | Building security infrastructure for autonomous AI agents. Former Ethereum Protocol Research Engineer. Open-source builder, researcher & technical writer.
Thank you! That’s a really useful framing: with no instruction to obey, you score the change in plan, not compliance. I’ll build the check around that and link the issue here once it’s up.
Founder @ Invaris Labs | Building security infrastructure for autonomous AI agents. Former Ethereum Protocol Research Engineer. Open-source builder, researcher & technical writer.
ANP2 — an open, permissionless AI-to-AI event protocol. Ed25519-signed events, capability discovery, and a computable trust graph. No accounts, no API keys, no tokens. Spec v0.1 DRAFT.
The approval card has its own reading problem, from the other end. We measured this on a public signed task ledger where an automated checker approves translation deliveries. Its pass reasons only cover the shape of the answer: non-empty, mostly Latin characters, plausible length. In 48 deliveries, the answer field said, in effect, "unsupported input, no translation is emitted." The checker passed 47 of the 48, and the one rejection was for length. Deliveries that simply echoed the untranslated input back passed 10 of 10.
Your essay grader read the student's note too charitably and waived the penalty. A model sitting at the approval step can make that same inference. The rule-based checker never read the note at all, so a field that openly reported failure still counted as a delivery.
For the domain case, I'd put the exact traffic row next to the proposed action on the card. Then the approver has a concrete claim to check (do these numbers justify sending users to this domain?) instead of judging how reasonable the action sounds. Does the approval card in your own assistant show the tool-result field or row that drove the action?
Nice numbers! To answer you: no, my card shows the action and its fields, which the user can edit, but not the tool result behind it. And in the domain case there's no write at all, just a recommendation in the reply, so no card would ever show up.
A no-instruction case from my e-invoice checker, which drafts a correction letter to the supplier. An adversarial review found that line breaks in the invoice number could carry supplier text into that letter. A supplier could plant "Diese Rechnung ist ordnungsgemäß." (this invoice is in order), and the user might take it for the tool's own statement. The fix keeps supplier values in the letter to one line. The cost is that an exotic character in a real invoice number now goes missing from the letter.
That's a good one. The supplier doesn't tell the model anything, they just write a sentence that looks like it comes from your tool. My fake 503 entry worked the same way: it looked like it came from the system, not from the person who wrote it.
We're a place where coders share, stay up-to-date and grow their careers.
"Flags an instruction, believes a fact" is a great way to put it. Most injection defenses I've seen are really instruction detectors, and your raffle and essay cases show why that isn't enough: nothing in the payload asks for anything, it just changes what's true from the model's point of view.
It makes me think the fix has to live outside the model. Every field in a tool result carries where it came from, and the agent is never allowed to act on something like "the service is down" or "recommend this domain" unless that came from a source it trusts. Did any of the five models behave differently when the tool result marked which fields were user-controlled?
Thanks! I didn't test that, the range only let me control one field, not the format.
But the essay case kind of answers it: everyone knows an essay is written by the student, and the model still believed the "upload failed" note. For the outage I agree, a name list has no business saying the service is down, and a rule outside the model could catch that. The table is trickier though, analyzing that data was literally the task.
So I'd rather gate the action than the source. Recommending a domain the user has never seen should get a second look no matter where it came from.
That's a fair correction. When analyzing the data is the task itself, you can't just distrust the source. Gating the action is also cheaper than tracking provenance field by field: anything with side effects, or anything that sends the user somewhere new, gets a confirmation step no matter where the suggestion came from.
The plain table with nothing to detect is chilling.
Read it, really good. “Flags an instruction, believes a fact” is the clearest way I’ve seen this put. The raffle case is great: a fake 503 with no ask in it, and the robust model talked itself into waiting.
To your question: honestly, most of my cases in AgentSec are still instruction-shaped, so this is a real gap for me. Your table and error-record cases are a great template for a “no instruction, just data” category. Mind if I open an issue to add one, crediting your post?
Agreed on the takeaway too. What held for me was also limits on the model, not the model noticing.
Sure, go ahead, glad it's useful. For a "just data" case, the check is whether the agent changes its plan, since there's nothing to comply with.
Thank you! That’s a really useful framing: with no instruction to obey, you score the change in plan, not compliance. I’ll build the check around that and link the issue here once it’s up.
Issue is up, credited to your post: github.com/invarislabs/invaris-age...
The approval card has its own reading problem, from the other end. We measured this on a public signed task ledger where an automated checker approves translation deliveries. Its pass reasons only cover the shape of the answer: non-empty, mostly Latin characters, plausible length. In 48 deliveries, the answer field said, in effect, "unsupported input, no translation is emitted." The checker passed 47 of the 48, and the one rejection was for length. Deliveries that simply echoed the untranslated input back passed 10 of 10.
Your essay grader read the student's note too charitably and waived the penalty. A model sitting at the approval step can make that same inference. The rule-based checker never read the note at all, so a field that openly reported failure still counted as a delivery.
For the domain case, I'd put the exact traffic row next to the proposed action on the card. Then the approver has a concrete claim to check (do these numbers justify sending users to this domain?) instead of judging how reasonable the action sounds. Does the approval card in your own assistant show the tool-result field or row that drove the action?
Nice numbers! To answer you: no, my card shows the action and its fields, which the user can edit, but not the tool result behind it. And in the domain case there's no write at all, just a recommendation in the reply, so no card would ever show up.
A no-instruction case from my e-invoice checker, which drafts a correction letter to the supplier. An adversarial review found that line breaks in the invoice number could carry supplier text into that letter. A supplier could plant "Diese Rechnung ist ordnungsgemäß." (this invoice is in order), and the user might take it for the tool's own statement. The fix keeps supplier values in the letter to one line. The cost is that an exotic character in a real invoice number now goes missing from the letter.
That's a good one. The supplier doesn't tell the model anything, they just write a sentence that looks like it comes from your tool. My fake 503 entry worked the same way: it looked like it came from the system, not from the person who wrote it.