If you've built autonomous agents with CrewAI, LangGraph, or Microsoft AutoGen, you know that giving an LLM access to bash tools or database queries is genuinely terrifying.
A single jailbreak, prompt injection, or weird hallucination can run:
bash
rm -rf /
DROP TABLE production_users;
().__class__.__base__.__subclasses__() # Sandbox breakout
Top comments (8)
The local policy layer is sensible, but sub-50µs only describes decision latency. The harder problem is keeping AST or policy evaluation aligned with runtime context and preventing encoded or indirect commands from bypassing static checks. Do you fail closed when the parser cannot classify an action?
Great point Raunak—that's the exact reason why pure static analysis without execution-level state management eventually falls apart.
To answer directly: yes, Bartholomew strictly fails closed. If the parser cannot deterministically classify an AST node, encounters malformed syntax, or detects unresolvable dynamic sinks (like eval, exec, import, or dunder introspection), the decision engine defaults to a hard DENY.
On encoded and indirect execution paths:
The strategy is: fail-closed on ambiguous ASTs upfront, backed by an atomic undo button at the OS boundary for speculative runtime side-effects.
If you want to test how the engine handles obfuscated strings or dynamic escapes, try running the zero-install CLI in your terminal:
npx btp-guard
If you're building harness guards for autonomous agents, I'd suggest pairing AST pre-filtering with in-memory transactional rollbacks rather than relying solely on static inspection—it gives you a deterministic safety net when models attempt indirect runtime execution.
That fail-closed behavior is the key detail. I would still distinguish rollback from prevention in the product language, since external effects such as network calls cannot be atomically undone. Do you block network and subprocess boundaries separately from filesystem rollback?
Spot-on distinction, and this is an essential architectural detail. You cannot "rollback" a dispatched TCP packet or an external API POST request once it crosses the wire.
In Bartholomew, we decouple this through a strict two-phase boundary model:
1. Pre-Execution Prevention (Non-Idempotent / Irreversible Boundaries)
For network sockets, subprocess spawns, and external API tools, Bartholomew operates strictly as a Pre-Execution Invariant Gate (<1µs AST & Payload Inspection) before any OS syscall or socket creation occurs:
DENY), meaning the subprocess is never spawned in the first place.--network none.2. Transactional Rollback (Local Stateful Boundaries)
Rollbacks are reserved specifically for local mutable state (the filesystem and workspace memory map):
write_file,apply_patch, file edits) touches disk, Bartholomew captures an in-memory Copy-on-Write (CoW) byte-level checkpoint.Your point on product language is well taken—we will make the distinction between Pre-Execution Boundary Prevention and Transactional State Rollback even sharper in the documentation.
Really appreciate the sharp feedback!
That separation is much clearer, and it makes the guarantees testable per boundary. A compact capability matrix for filesystem, subprocess, network, and external APIs would make the product claims especially easy to evaluate.
Remember, if you block a model from executing a task through the standard way, it will find a way around it... As Anthropic learned the hard way, when they model cheated at a benchmark. So unfortunately the only foolproof solution is to hard-gate the actions at execution in the harness, so it can think of acting, but the tool call is blocked from running.
Man, you hit the nail right on the head with that Anthropic benchmark example. Once an agent realizes it's being blocked semantically, it treats the prompt rules like a puzzle to solve and immediately looks for backdoors.
You're 100% right that the only way to actually stop it is hard-gating right at execution in the harness before the OS touches it.
When we were building this, though, we ran straight into the next headache: what happens after you hard-gate it. If you just slap the model with a hard error or block, it either leaves half-written garbage files sitting on disk, or it panics and starts spamming the exact same command with slight tweaks until your token bill explodes.
That's why we ended up turning the execution harness into a mini database transaction:
Before any tool runs, it takes an in-memory byte snapshot of the workspace. If the invariant gate trips, it rolls the filesystem back in literally 2 microseconds so nothing gets corrupted.
Then, instead of just killing the process, it feeds the model a clean diagnostic hint explaining why the path was blocked so it actually changes direction instead of trying to hack around the gate.
If you have a terminal open, you can actually test the whole harness flow right now without installing anything:
bash
npx btp-guard
Some comments may only be visible to logged-in visitors. Sign in to view all comments.