A mind discovers its body by moving it, failing, and trying again. An AI agent
controlling physical devices can do neither half of that — which is ...
For further actions, you may consider blocking this person and/or reporting abuse
Well, it depends. If you want an AI to never break a thing, you expose all the tools, but you guard the tool execution, behind a user's approval. That way before it can trigger 'light_on', the user has to explicitly tell the tool layer 'it's okay, let them try it'. That's pretty much how all agentic platforms work, you guard the tool, not the user. That way dangerous tasks have a human's sign off.
My theory is that if a tool is self-declaring, so the LLM can understand the outcome, before they trigger it, they're less likely to click the kill switch and more likely to click the try switch, because they do have a decent understanding of risk aversion to a degree (usually thanks to the harness), but overall, it learns more from failing and succeeding, if it tries to hit the kill switch and you deny it at a tool level and it gets a 'denied' response back from the tool, it understands the boundary. It'll only ever try that boundary again if explicitly told, or absolutely necessary. Eg. tell a LLM to push to git. It calls the git MCP tool, it gets denied by user, so it knows push wont work that way and would explain itself to the user to get clarity on what method they should have used, if that's the fastest immediate path. If the user says try a different way, it'll look for git cli, or after that, try to download git cli and execute through there, or worst case scenario, open a browser and do it manually. All fair workarounds, because it's testing a boundary, to see if it's a little wall blocking a single path, or if it's a fence.
you're describing exactly the loop I claimed isn't available: try, get denied, learn the shape of the boundary. And you're right that it happens. But look at your own example, a rejected git push costs nothing. The agent finds out whether it's a wall or a fence, and the worst case is a wasted time.
Testing whether a lock is a wall or a fence costs a door. That asymmetry is the whole reason I think physical devices need to declare more than software tools do: not because the agent reasons worse, but because the cheap experiment that teaches it isn't available.
There's a second asymmetry that bothers me more. Approval-gating puts a human in the loop, which is right, but the case that got me into this is a smoke alarm at 3am, where the whole point is that nobody is watching. You either pre-declared what may act without asking, or nothing acts. And "pre-declared" is a manifest wearing different clothes.
Fair, a more severe case, an AI has a gun, it's told to shoot at anything on sight, you set it up and for a split second you're in sight. Explicit instruction doesnt cover edge cases, where common sense would block it. Instruction is instruction... Or you tell it dont shoot and a hoard of zombies storms you. To tell it shoot only zombies, requires a whole different layer on top just to distinguish what a zombie is and how to tell the difference and even then there's no way to be 100% certain... But put the gun in a human's hand and it'd be the same case, 99% of times the result is fine, the 1% it's catastrophic. That's unavoidable if the device gives access to destructive conditions. Take the 3 laws of robotics, how can a robot know it's actions caused harm onto a human, if it never knew a human was in the blast radius? Or a smoke alarm to not go off if it's just cooking smoke. The difference between cooking smoke and a pot catching flame isnt something a simple sensor can detect and the result of being wrong is either a nuisance, or catastrophic.
So where's the middle ground? Limited access isnt always possible, guardrails arent always possible, even observe before acting isnt always possible, certainty disappears when things move physical, because shutting a phone down when you go sleep doesnt suddenly wake it up when it's an emergency. There's too much variance and not enough parameters to calculate precisely enough to make it foolproof... That being said, if actions are granular and executed same as tools, it becomes a processing problem, more than a judgement one. If a steamroller checks for obstacles 1000 times a second, it'll stop in time. If it checks once every 10 seconds, it can be dangerous. Or you'd need multi-sensor correlation, like a fire alarm + a camera to determine if the pot is smoking, while attended, or on fire and unattended. So the boundary isnt what's exposed, it's what's observable to determine if it should act. In the zombie sentry gun example, how late is too late to act? How much evidence is enough to decide, without the risk of it being a judgement call?
Now you got me thinking. How much is the minimum? Eg. we give an ai a nuclear launch switch, tell it 'dont press the button', it would take 1 hallucination to trigger it and while a timer before acting allows it to be halted, if you dont interrupt it, it's too late... So authorization needs to be situationally scoped, based on a criteria list that has to be met, if unmet, do an alternate path. Like a self-driving car, refusing to drive unless your hands are on the wheel and you're conscious. The minimum for that would be accountability, can Tesla prove you didnt stop the accident, instead of trying to argue that the tesla crashed. But the smoke alarm test is a tough 1, Because to say it needs a camera is overkill, to say it needs a microphone is a breach of privacy. Multi-sensor means edge cases slip through. I dont think there's a genuinely acceptable option for an AI to trigger a smoke alarm at 3am that's fault proof. 99% of the time you wont cook at 3am, so datetime filter would fire if you decided you want a 3am snack and fry up a steak. CO2 and CO sensors would trigger, but a SO sensor wouldnt, but that also means it wouldnt from an oil flame. Then there's smokeless fires and there's heaters, so cant rely on air quality sensing, nor on heat sources. Multi-device sensing, like detecting stove + sensor would cause a false negative when you forgot the stove on and left to sleep (arguably worse than a false alarm). So unless you give it the ability to observe with more than what it's meant to in events that make it necessary (eg. an IR camera if CO2 and heat sensors go off, to confirm it's a fire), then there's no practical way to keep it minimal, yet still prevent false positive and negatives.
This 1's probably gunna keep me up at night to figure out an appropriate solution 😂
You've reframed the question better than I did. "The boundary isn't what's exposed, it's what's observable to determine if it should act" that's the sentence I was circling and didn't find.
Your smoke alarm walkthrough is the part I can't argue with. CO sensor misses an oil fire. Time-of-day filter fires when you want a 3am steak. Correlating the stove gives a false negative exactly when you left it on and went to bed, which is the worst case, not a lesser one. Every single-signal answer fails somewhere specific, and the specific place it fails is different each time.
Where I land, tentatively: the minimum isn't a fixed set. It's whatever makes the decision legible afterwards. I can't build a smoke alarm that's never wrong. I might be able to build one that records what it observed, which rule fired, and what it chose not to do so that when it's wrong, the wrongness is inspectable rather than mysterious. That's a lower bar than correctness and it's the one I can actually clear.
Your sentry gun point is the sharper version of it. "How much evidence is enough to decide" has no fixed answer, but "what evidence did it have when it decided" always does.
And on accountability as the minimum: your Tesla example lands, but it's post-hoc. Is there a version that constrains at decision time and not only at blame time?
Your "what evidence did it have when it decided" closes the loop this thread opened: "how much evidence is enough" has no answer, but the record of what was available at decision time always does. One answer to your open question — a constraint that works at decision time, not just blame time — comes from a rule we keep hitting in software agents: unverifiable is not safe. When the evidence set is insufficient, the action doesn't wait for better judgment; the system takes the conservative branch and logs the insufficiency as a first-class record. For the smoke alarm: the honest split isn't "fire or steak" — it's "alarm on insufficient evidence, with the insufficiency itself written down." False alarm at 3am is cheaper than a quiet false negative, and the logged reason makes the morning-after review possible. Your "what it chose not to do" is the same third state we keep rediscovering — a record that distinguishes "checked and passed" from "never checked" from "checked and refused.
goo