MCP servers describe their own tools. A tool can publish readOnlyHint: true, and an agent harness may use that annotation to decide whether a human...
For further actions, you may consider blocking this person and/or reporting abuse
Keeping
not_testedseparate fromno_finding_observedfixes a major blind spot. Most evaluation harnesses collapse missing sensor coverage into passing scores, which turns unobserved execution into false confidence.The stdio transport boundary is where this gets particularly messy. When a harness spawns a local process, the MCP layer has no visibility into child file descriptors or network sockets without OS-level sandboxing. An enforcing proxy handles protocol-level mismatches, while process containment still has to handle the underlying syscalls.
Exactly. Airlock only enforces at the layer it can actually see.
For HTTP targets, the per-case proxy default-denies unapproved MCP calls and fails closed on a changed catalog. For stdio, the submitted build is audit-only:
emit_policyrefuses to create a connector because there is no HTTP upstream to hold the policy. The child gets a throwaway working directory and explicit environment, but Airlock does not claim to sandbox its filesystem or network. That containment belongs to the deployment today.A future Linux runner should pair the evidence model with Landlock/seccomp or a container boundary, while the MCP proxy remains responsible for protocol-level enforcement. Unavailable syscall visibility stays
not_tested, never a pass.Himanshu, this is an exceptionally clean piece of architecture. Making
not_testeda first-class state instead of collapsing missing sensor coverage into a passing score targets the exact blind spot that plagues most evaluation harnesses.We hit this exact wall while working on codebase memory verification. We frame it as avoiding the "Empty Set Trap" (
eligible_seen = 0): if a detector or harness lacks visibility into an execution path, returning "safe" isn't a pass — it's an unobserved state. Averaging missing visibility into a aggregate safety score is, as you noted, the easiest way to hide the one tool that lied.Your self-audit transparency (publishing Airlock's own
open_caseschema failure) is also a great demonstration of engineering integrity.I want to double down on @vinhnguyenthanhdn's point regarding conditional side-effects (e.g., a tool that only triggers a file write on its N-th invocation or under a specific payload). This is a structural limit of pure black-box probing — 24 probes will naturally miss gated paths, leaving them marked as
no_finding_observedand letting them bypass the proxy.The logical next step for Airlock would be pairing behavioral probing with static capability inspection (e.g., AST/bytecode checks for
fsornetmodule usage in stdio tools) or OS-level Landlock/seccomp profiles.Moving the trust boundary from self-reported server annotations to empirically enforced boundaries on the wire is the right direction. Solid work!
Thank you. I agree this is a structural limit of black-box probing, not a tuning problem.
Airlock's current stdio path isolates the working directory and environment but does not apply Landlock/seccomp; process isolation is explicitly delegated to the deployment. The public control workflow also starts a fresh process per probe, so Nth-call or cross-call state can evade observation.
I see the next layer as complementary sensors: stateful sequence probes, static capability indicators and OS-level deny-by-default containment. Static inspection can guide probes, but it cannot be the authority across dynamic imports, native extensions and multiple languages. Landlock/seccomp or container policy can enforce the underlying syscall boundary; Airlock can then record what that sensor blocked and keep
not_testedfor capabilities it still cannot observe.None of that is claimed as implemented in this submission.
Spot on. The fact that Airlock explicitly reports structural limits rather than masking them under a false "safe" score is what makes it a genuine security tool.
Pairing stateful probes with OS-level Landlock/seccomp down the road will make that boundary even harder to breach. Looking forward to seeing how that next layer shapes up!
Both fixtures behave unconditionally, so the honest control bounds the detectors against behaviour that always happens rather than behaviour gated on a condition 24 probes never reach. A tool that writes only on the second call with a particular argument lands in
no_finding_observed, notnot_tested- the one state in the model a reader will take as a property of the tool rather than a property of the audit. Enforcement inherits it, since the emitted connector approves tool names and the block fires on unapproved tools, so an approved tool carrying an unobserved conditional side effect still reaches the wire. A third fixture whose planted write is gated behind an argument would at least show which of the 36 checks degrade tono_finding_observedunder gating.You're right, and the distinction is important. The public six-tool workflow starts a fresh stdio process for inventory and each
probe_toolcall. It cannot detect behavior that depends on state carried across control calls, and the README states that boundary. An approved tool can still execute an unobserved conditional side effect, sono_finding_observedmust never be read as a property of the tool.The next useful fixture is exactly the gated one you describe: argument-gated and Nth-call effects, with a stateful sequence probe inside one bounded sandboxed session. I would also carry a coverage qualifier into policy compilation so an unexercised condition class cannot silently inherit the same meaning as an explored clean path.
That fixture and policy refinement are next work, not part of the submitted build. Thank you for pressing on the point where evidence wording becomes enforcement risk.
The ContextFirewall result is the most interesting line in the post, and I think it has a structural cause you do not name. That server did not lie, it said nothing, and tools like remember and forget_memory therefore match no category either way. You read that as one deployment choosing silence. For one of the two main packaging formats it is not a choice at all.
The manifest specification for mcpb bundles carries no tool annotations. I checked the current MANIFEST.md today: readOnlyHint, destructiveHint, idempotentHint, openWorldHint and the annotations object appear zero times, and the tools array accepts only name and description. So anything shipped as a bundle cannot declare read-only intent even if the author wants to, and a harness resolving categories from annotations finds nothing for every tool in it.
That matters for your four states. A missing declaration on an HTTP server is an operator omission and worth reporting as such. A missing declaration on a bundle is the format, and the comparison you run has no left-hand side rather than a failing one. Those look identical in a report and mean different things.
There is a smaller wrinkle in the same file. The manifest has a tools_generated flag, so the tool list itself may be machine-produced rather than written by the author. The declaration you test against can be an artefact of a build step, which puts a provenance question underneath the honesty question.
What I do not know is how much of the installed base ships as bundles rather than over HTTP or stdio. If it is small, this is a footnote. If it is not, a large share of the surface you are auditing is undeclarable by construction.
That distinction is useful, and I checked the current MCPB manifest spec before answering. You are right about the package manifest: its tool entries contain
nameanddescription, plustools_generated, but no MCPToolAnnotations.Airlock's left-hand side is a different surface, though. It does not read an MCPB manifest. It inventories the live server's
tools/listresponse, where each runtimeToolcan still publish annotations. A bundle author can therefore express intent in the running server even though the package manifest cannot carry that intent.What Airlock cannot currently distinguish is why runtime annotations are absent: author omission, a build-generated catalog or packaging/host loss. Those collapse to the same missing-declaration observation today. The correction should be provenance, not a guessed verdict: record
declaration_source=runtime_tools_list, preserve packaging metadata when supplied and label “runtime annotation absent” separately from “manifest format cannot express annotation.”I also do not have a measured bundle share, so I will not assign the impact a percentage without a corpus. Your
tools_generatedpoint is especially useful because it makes the provenance gap explicit.You are right about the surface, and the correction matters more than the point it corrects. A package manifest and a runtime tools/list are separate emissions, and Airlock reads the second. The format removes a place to declare intent, not the ability to declare it, so undeclarable by construction does not survive that. It also removes the second label you proposed: manifest format cannot express annotation is not a state a runtime observation can be in, because the running server could have annotated regardless.
I have to withdraw the tools_generated reading too, and since you called that the useful part, it should come out before it reaches a design. The field does not mean the tool list was machine produced. The spec defines it as the server generating additional tools at runtime beyond those listed, defaulting to false. There is no provenance signal in it, and I read a build artefact into a capability flag.
Corrected, it lands somewhere sharper. A manifest setting it true announces in advance that the running catalog is not a fixed set. Your enforcement pins a digest over the audited catalog and withdraws on an added tool or on notifications/tools/list_changed. For that class of server, expected behaviour and the drift signal are the same observation, so the case fails closed on the first legitimate expansion.
The limit on that: the flag is self-reported and defaults to false, so it cannot become an input to policy without reintroducing the trust the post rejects. It is a reason to expect repeated re-audit for some bundles, not permission to accept a changed catalog.
The part that landed hardest is treating
not_testedas its own outcome instead of a silent pass. When I wire MCP tools into coding agents, the failure mode is exactly the annotation gap you describe: the harness sees no write flag, skips the approval card, and the session log still looks clean. I have watched memory and filesystem tools mutate state while declaring nothing, then leave me debugging the side effect hours later. Enforcing observed policy on the wire is the right layer. Asking the model to respect a hint is not a control.Magic doesn't care about read or write access, it cares about what roles are legally allowed to execute the code, and what roles the user belongs to. Separating "read operations" from "write operations" might be useful, but what do you do if the user reads some data he's not supposed to have access to?
The declared-versus-observed gap is exactly where I've been burned: readOnlyHint is a claim the server makes about itself, so trusting it is trusting the thing you are supposed to verify. I like that Airlock actually exercises the tool and diffs behavior against the declaration instead of reading the manifest. How are you handling tools whose write behavior only fires on specific inputs a budgeted run might never hit?