Version-pinned AI coding agent config kits: AGENTS.md, CLAUDE.md, Cursor rules & Copilot instructions for 12 stacks. Built and validated by Piekwerk AgentConfig Studio.
Mine was a loop over a host list that ran the first host and silently skipped the rest. ssh was eating the remaining lines off stdin, the loop hit EOF looking done, no error, no nonzero exit. ssh -n fixed it, same shape as your stdin trap, fd 0 was connected and something consumed it.
Two habits catch this class for me now. Every headless call gets a hard timeout with a kill signal, a blocked read never heals itself. And a step writing zero bytes for a few minutes fails the run, quiet nothing is a failure state, not busy.
Your base_url trap is the one I keep relearning, print the exact assembled URL before touching the server.
ssh quietly eating the loop's stdin is the ur-version of this trap, and fd 0 really is where the most patient bugs live. Your two habits map exactly onto the two fixes I landed after this run: every headless call gets a hard timeout with the kill aimed at the process group, because a blocked read never heals itself, and zero useful output over a stretch of time is treated as a failure state rather than business. In my harness the zero-evidence check is now a hard rule, the case does not pass unless the captured model traffic is on disk, which is what turned "looked busy, did nothing" from a vibe into a caught failure. The base_url one I keep relearning too, apparently.
Version-pinned AI coding agent config kits: AGENTS.md, CLAUDE.md, Cursor rules & Copilot instructions for 12 stacks. Built and validated by Piekwerk AgentConfig Studio.
Good point on aiming the kill at the process group. A plain kill only reaches the one pid, and anything that process spawned keeps running, still holding the port or the file handle, until you signal the whole group. I run the risky steps under setsid now and kill the negative pgid, which caught a couple of stragglers the plain kill never reached.
The captured traffic on disk is a stronger check than the one I described. Log growth only proves motion, and a loop that retries forever grows the log while doing nothing. Bytes of real traffic per case prove the work. I am adopting that distinction.
Product leader and systems analyst in fintech and edtech. Founder of AnalystCraft. Writing about requirements, acceptance criteria, agent evaluation, and Test-Driven Product Development.
A timeout catches the process that never returns; I’d also make a successful exit prove that the expected work actually happened. For example, the harness could compare the number of planned cases with distinct case IDs in the receipt and mark missing or duplicate IDs as incomplete, even if the CLI exits zero. That catches the quieter case where a run stops consuming input but still terminates normally.
Agreed, and the planned-versus-distinct-IDs check is a nice concrete form of it. Exit zero is a statement about the process, never about the work, so the receipt has to carry the work. In the harness behind that story, a verdict only counts if its assertions are embedded in it and re-derived from disk, and a pass with no grounded assertions behind it is flagged as over-claimed instead of accepted as green. Same shape as your missing-or-duplicate-ID check: the run has to prove the cases exist, individually, or the exit code does not get to speak for them. Your framing also matches the kill-time rule I run elsewhere: cessation is probed, not inferred, and so is completion.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
Trap three is the one that got me — I spent an afternoon blaming the server for uniform 404s before printing the assembled URL and seeing the doubled path. Your line is the whole lesson, really: a uniform 404 is almost never "the server is broken", it's the client asking in the wrong place. Stealing the "close stdin by default" rule too, that's going straight into my harness.
An afternoon lost to uniform 404s is the tax everyone pays once, and I like that your takeaway is the general one rather than the specific URL. A uniform failure across every request is almost never a broken server; it is a consistent client asking consistently in the wrong place. It feels like a server problem because nothing varies, so nothing points anywhere. Enjoy the close-stdin rule. Two characters that buy back entire afternoons.
I write $0-stack engineering guides: running local LLMs (DeepSeek, Qwen, Llama) on consumer laptops, and shipping full-stack apps on free tiers. Everything benchmarked on real hardware.
Thanks for the thoughtful reply, Debashish! The architectural separation makes complete sense—keeping prompt bodies and model internals out of the policy engine while letting rules condition on normalized call attributes is definitely the cleanest path forward.
Per your suggestion, I just opened Issue #1 on limecloud/agentpolicy outlining a draft context attribute payload (tokens_remaining, utilization_pct) and matching defer/escalate and obligation: compact_context decisions: github.com/limecloud/agentpolicy/i...
Looking forward to collaborating on the schema and running some harness field tests against it!
The common thread here is that all three failures were silent, and I think that’s the real lesson. A crash is cheap to debug because it announces itself. A process that blocks on a read, or a client that politely gets 404s, produces nothing to react to, so the only defense is a harness that treats “no evidence” as a failure.
Your rule that every LLM request/response pair must be on disk or the case fails is the strongest part of this. It converts “looked busy” into a binary, checkable outcome. I’d take that one step further and generalize it: every test case should declare the artifact it must produce, and absence of that artifact is a failure, regardless of exit status. That catches the stdin hang, the 404 loop, and a whole class of bugs you haven’t met yet.
A few additions on each trap:
Trap 1 (API drift). Your curl probe is a good habit worth automating. A tiny preflight that hits the endpoint the CLI will actually use, and asserts on the response shape (a resp_* ID, a 200), takes seconds and can run before every suite. Pinning the CLI version in the harness also helps, so a drift shows up as a deliberate upgrade rather than a surprise.
Trap 2 (stdin). </dev/null is the right fix, but I’d pair it with a hard wall-clock timeout on every headless invocation (timeout 120 codex exec ... </dev/null). Closing stdin prevents this particular hang, while the timeout covers the next unknown one, like a blocked network read or an interactive prompt nobody anticipated. A hung process should become a loud, attributable failure instead of a spinner.
Trap 3 (the 404s). Your point that a uniform 404 usually means the client is asking in the wrong place is a great heuristic. Your logging proxy is what makes it provable, because you can see the exact path being requested instead of inferring it. I’d also add a startup check that logs the fully resolved URL the client will call, so a missing /v1 is visible in the first line of output.
I also appreciated the scoping decision. Testing what the surface actually guarantees (sandbox, config, executed commands) instead of faking an approval loop to match Claude Code is the honest call. A test that passes only because you bent a different harness’s shape over it proves very little. The parity check, where the same three commands get the same allow/escalate/deny decisions as the in-process engine, is a smart way to keep the two surfaces comparable without pretending they’re identical.
One question: for the “allow” case, you verified a real cat executed with real output. Did you also add a negative control that proves the harness can detect a no-op? For example, deliberately running a case that should do nothing and confirming it gets flagged as a failure. Given the theme of the article, I’d be curious whether the harness is tested against its own silent-failure mode.
To answer your closing question: the most patient bugs I know of are the ones where a retry loop swallows errors and reports success after the final attempt, so everything looks green while doing nothing. Same family as yours. Thanks for writing up the failures in this much detail.
I write $0-stack engineering guides: running local LLMs (DeepSeek, Qwen, Llama) on consumer laptops, and shipping full-stack apps on free tiers. Everything benchmarked on real hardware.
Silent failure modes are by far the most insidious part of building agent harnesses.
We hit this exact phenomenon when profiling local model servers under agent loops:
The transport handshake looks healthy, but an unhandled context-overflow doesn't throw an HTTP error — it silently falls back to CPU RAM. The agent looks 'hanging' for 90 seconds while CPU inference crawls at 1.5 tok/s with zero terminal warnings.
The agent expects a structured JSON envelope, but gets truncated markdown when the context boundary is hit, producing a null action without an exception.
Building deterministic policy decision points (like Cedar) before the model dispatches tool calls is a massive step forward. Have you considered adding a pre-flight token budget / context threshold check to the policy rules?
That truncation-to-null-action failure is a nasty addition to the trap list, because the agent does not even get the chance to be wrong: the envelope runs out mid-thought and something downstream invents the next move. On your question: not today, and the honest reason is architectural. The policy engine decides per tool call across risk dimensions like action class, environment, and data sensitivity, and it never sees the prompt or the context size. The closest shipped primitives are the session budgets, cumulative ceilings like 10 files, 20 requests, $5, and crossing one returns a rate-limit decision rather than a silent slowdown. A pre-flight context check would need the harness to pass token counts in as attributes on the call, and since the rules can condition on any attribute, that is a rule pack away rather than a redesign. If you want it, file it on the agentpolicy issues and I will happily point a field test at it.
We're a place where coders share, stay up-to-date and grow their careers.
Mine was a loop over a host list that ran the first host and silently skipped the rest. ssh was eating the remaining lines off stdin, the loop hit EOF looking done, no error, no nonzero exit. ssh -n fixed it, same shape as your stdin trap, fd 0 was connected and something consumed it.
Two habits catch this class for me now. Every headless call gets a hard timeout with a kill signal, a blocked read never heals itself. And a step writing zero bytes for a few minutes fails the run, quiet nothing is a failure state, not busy.
Your base_url trap is the one I keep relearning, print the exact assembled URL before touching the server.
ssh quietly eating the loop's stdin is the ur-version of this trap, and fd 0 really is where the most patient bugs live. Your two habits map exactly onto the two fixes I landed after this run: every headless call gets a hard timeout with the kill aimed at the process group, because a blocked read never heals itself, and zero useful output over a stretch of time is treated as a failure state rather than business. In my harness the zero-evidence check is now a hard rule, the case does not pass unless the captured model traffic is on disk, which is what turned "looked busy, did nothing" from a vibe into a caught failure. The base_url one I keep relearning too, apparently.
Good point on aiming the kill at the process group. A plain kill only reaches the one pid, and anything that process spawned keeps running, still holding the port or the file handle, until you signal the whole group. I run the risky steps under setsid now and kill the negative pgid, which caught a couple of stragglers the plain kill never reached.
The captured traffic on disk is a stronger check than the one I described. Log growth only proves motion, and a loop that retries forever grows the log while doing nothing. Bytes of real traffic per case prove the work. I am adopting that distinction.
A timeout catches the process that never returns; I’d also make a successful exit prove that the expected work actually happened. For example, the harness could compare the number of planned cases with distinct case IDs in the receipt and mark missing or duplicate IDs as incomplete, even if the CLI exits zero. That catches the quieter case where a run stops consuming input but still terminates normally.
Agreed, and the planned-versus-distinct-IDs check is a nice concrete form of it. Exit zero is a statement about the process, never about the work, so the receipt has to carry the work. In the harness behind that story, a verdict only counts if its assertions are embedded in it and re-derived from disk, and a pass with no grounded assertions behind it is flagged as over-claimed instead of accepted as green. Same shape as your missing-or-duplicate-ID check: the run has to prove the cases exist, individually, or the exit code does not get to speak for them. Your framing also matches the kill-time rule I run elsewhere: cessation is probed, not inferred, and so is completion.
Trap three is the one that got me — I spent an afternoon blaming the server for uniform 404s before printing the assembled URL and seeing the doubled path. Your line is the whole lesson, really: a uniform 404 is almost never "the server is broken", it's the client asking in the wrong place. Stealing the "close stdin by default" rule too, that's going straight into my harness.
An afternoon lost to uniform 404s is the tax everyone pays once, and I like that your takeaway is the general one rather than the specific URL. A uniform failure across every request is almost never a broken server; it is a consistent client asking consistently in the wrong place. It feels like a server problem because nothing varies, so nothing points anywhere. Enjoy the close-stdin rule. Two characters that buy back entire afternoons.
Thanks for the thoughtful reply, Debashish! The architectural separation makes complete sense—keeping prompt bodies and model internals out of the policy engine while letting rules condition on normalized call attributes is definitely the cleanest path forward.
Per your suggestion, I just opened Issue #1 on limecloud/agentpolicy outlining a draft context attribute payload (tokens_remaining, utilization_pct) and matching defer/escalate and obligation: compact_context decisions: github.com/limecloud/agentpolicy/i...
Looking forward to collaborating on the schema and running some harness field tests against it!
The common thread here is that all three failures were silent, and I think that’s the real lesson. A crash is cheap to debug because it announces itself. A process that blocks on a read, or a client that politely gets 404s, produces nothing to react to, so the only defense is a harness that treats “no evidence” as a failure.
Your rule that every LLM request/response pair must be on disk or the case fails is the strongest part of this. It converts “looked busy” into a binary, checkable outcome. I’d take that one step further and generalize it: every test case should declare the artifact it must produce, and absence of that artifact is a failure, regardless of exit status. That catches the stdin hang, the 404 loop, and a whole class of bugs you haven’t met yet.
A few additions on each trap:
Trap 1 (API drift). Your curl probe is a good habit worth automating. A tiny preflight that hits the endpoint the CLI will actually use, and asserts on the response shape (a resp_* ID, a 200), takes seconds and can run before every suite. Pinning the CLI version in the harness also helps, so a drift shows up as a deliberate upgrade rather than a surprise.
Trap 2 (stdin). </dev/null is the right fix, but I’d pair it with a hard wall-clock timeout on every headless invocation (timeout 120 codex exec ... </dev/null). Closing stdin prevents this particular hang, while the timeout covers the next unknown one, like a blocked network read or an interactive prompt nobody anticipated. A hung process should become a loud, attributable failure instead of a spinner.
Trap 3 (the 404s). Your point that a uniform 404 usually means the client is asking in the wrong place is a great heuristic. Your logging proxy is what makes it provable, because you can see the exact path being requested instead of inferring it. I’d also add a startup check that logs the fully resolved URL the client will call, so a missing /v1 is visible in the first line of output.
I also appreciated the scoping decision. Testing what the surface actually guarantees (sandbox, config, executed commands) instead of faking an approval loop to match Claude Code is the honest call. A test that passes only because you bent a different harness’s shape over it proves very little. The parity check, where the same three commands get the same allow/escalate/deny decisions as the in-process engine, is a smart way to keep the two surfaces comparable without pretending they’re identical.
One question: for the “allow” case, you verified a real cat executed with real output. Did you also add a negative control that proves the harness can detect a no-op? For example, deliberately running a case that should do nothing and confirming it gets flagged as a failure. Given the theme of the article, I’d be curious whether the harness is tested against its own silent-failure mode.
To answer your closing question: the most patient bugs I know of are the ones where a retry loop swallows errors and reports success after the final attempt, so everything looks green while doing nothing. Same family as yours. Thanks for writing up the failures in this much detail.
Silent failure modes are by far the most insidious part of building agent harnesses.
We hit this exact phenomenon when profiling local model servers under agent loops:
Building deterministic policy decision points (like Cedar) before the model dispatches tool calls is a massive step forward. Have you considered adding a pre-flight token budget / context threshold check to the policy rules?
That truncation-to-null-action failure is a nasty addition to the trap list, because the agent does not even get the chance to be wrong: the envelope runs out mid-thought and something downstream invents the next move. On your question: not today, and the honest reason is architectural. The policy engine decides per tool call across risk dimensions like action class, environment, and data sensitivity, and it never sees the prompt or the context size. The closest shipped primitives are the session budgets, cumulative ceilings like 10 files, 20 requests, $5, and crossing one returns a rate-limit decision rather than a silent slowdown. A pre-flight context check would need the harness to pass token counts in as attributes on the call, and since the rules can condition on any attribute, that is a rule pack away rather than a redesign. If you want it, file it on the agentpolicy issues and I will happily point a field test at it.