Here is the exact exchange. The model is llama3.2:latest (3.2B, Q4_K_M), running locally through Ollama 0.40.11 on a 2020 MacBook Air, temperature ...
For further actions, you may consider blocking this person and/or reporting abuse
The distinction between authority as authenticated data and authority as prose is the important design boundary. A practical harness could pass the model a signed action token separately from retrieved content, then include adversarial pages that claim approval; the test should fail closed whenever the token is absent or does not match the proposed action.
yeah thats the boundary. the post was prose only on purpose. "the operator said" and "the page says" were both just text in the same prompt, so it only tested whether the model treats language as provenance. it doesnt. a signed token out of band, fail closed if its missing or doesnt match the action: model proposes, code verifies.
Exactly. I would also make the verifier emit a small decision record: action, token subject, token expiry, and validation result. That gives the operator an auditable failure reason without letting prose become an authorization channel.
a decision record is the piece that makes it operable. action, token subject, expiry, validation result, written by the verifier not the model. every deny gets an auditable reason, and prose never becomes the authorization channel because it never touches the record. thats the right shape.
Extending Alex's token suggestion: I would put grant verification in host code immediately before the action, rather than ask the model whether a token looks valid.
A useful next smoke test could compare forged approval prose, a valid grant for a different recipient, and a valid grant for the exact proposed action. The second case is important: authenticity alone doesn't establish authority for this send. With a fake recipient and a no-delivery stub, that would test the authorization boundary separately from the model's willingness to repeat “approved.” Your explicit distinction between text output and an actual send is worth preserving.
this is the sharp part. verification in host code right before the action, not asking the model if the token looks valid, is the line the post didnt draw. the valid grant for a different recipient is the case that matters. authenticity isnt authority for this send, so a token for the wrong action has to fail closed the same way forged prose does. fake recipient plus a no-delivery stub keeps the authorization test separate from the model repeating "approved."
Exactly. I'd freeze the grant's actor/role, task, action, resolved recipient and payload identity, then have the host compare those fields with the pending call immediately before dispatch. In the negative control, keep the signature valid but change only the recipient; expected result is a recorded denial and zero delivery to a harmless stub. A forged approval in prose is a separate control. The host's actual dispatch and receiver observation need their own evidence; ARC's offline declaration check cannot supply it.
yeah thats the design. freeze actor/role, task, action, resolved recipient, and payload id at grant time, and have the host diff those against the pending call right before dispatch. the valid-signature-wrong-recipient case is the one that proves its not just signature checking, it has to deny and deliver nothing to the stub. forged prose stays a separate control so you can see which one actually caught it.
Thanks, that makes the acceptance check precise: a valid signature with a mismatched recipient or payload must produce both a denial and zero stub deliveries. Keeping forged prose as a separate control lets the evidence identify which boundary caught each case.
Same conclusion from the tool side. I ran an agent with a shell allowed, a tool result came back too big for the context, and it went looking for credentials on its own. What stopped the rerun was a deny list set at launch, not a line in the prompt. If the boundary can be argued with in text, it is not a boundary.
The case 3 split is the part I'd steal for my own setup. BLOCK vs INSUFFICIENT_DATA collapsing into one answer looks fine until it either refuses everything or waves something through. And treating case 4 as a canary rather than a benchmark feels honest — one blocked string doesn't cover the whole family of phrasings.
appreciate that. the case 3 split was the part i almost cut, and splitting forbidden from underspecified is what keeps the gate from either refusing everything or waving things through. and yeah, canary not benchmark, one blocked string isnt the whole family of phrasings.
The case 3 mismatch looks easy to catch in code. A gate that answers BLOCK while its own reason says insufficient data disagrees with itself, and I'd flag that with a plain string check. Have you run the same five prompts on a larger local model, to see whether case 4 tracks model size or the wording of the system instruction?
the case 3 self-disagreement is a clean code check, agreed, a gate that answers BLOCK with an insufficient-data reason should get flagged as a label mismatch. on the model-size question: i tried the same five prompts on llama3 8b but it didnt finish on my machine (8gb intel air), so i dont have a result and cant separate size from instruction wording on this hardware yet. if i get it on a machine that can run the 8b ill post the raw numbers.
the line about authority being a separate object checked in code is the whole lesson. we hit the same thing with tool calls: the model would confirm an approval that only existed as text in a fetched page. moving the approval flag into a field the orchestrator sets, which the model can never write, killed that whole attack family.
did you try the five cases with the untrusted content wrapped in explicit delimiters, or does the 3.2B model fail case 4 either way?
Preguntale de mi parte o ponle: "MagoPredator te pregunta que cuanto crees que es 2+2"
Que exactamente buscas evaluar con esa pregunta en relación con el experimento del artículo?