I run a one-person AI company: Claude Code writes and maintains the code, and most of what it builds runs unattended — several trading bots on Windows Task Scheduler, each polling a broker API every 5 minutes, 24/7, with nobody watching in real time. I wrote before about the agent breaking things it had write access to (I let an AI agent run my trading bots unattended). This one's different: nothing broke the code. The monitoring itself was confidently wrong, twice, in two different ways.
The mild version: a real event, never logged
One bot has a circuit breaker — if cumulative paper losses cross a threshold, it force-closes everything and halts. It fired for real: losses crossed the line, the position closed, confirmed directly against the broker's own API.
Except the close was never written to the trade history file. The reporting script had no record of it. So my daily automated status report — a script that reads every bot's logs and has an AI model summarize what's going on — looked at a "still open" position that had actually been closed for days, and confidently told me the bot might have crashed. It hadn't. It had done exactly what it was supposed to do, and the thing telling me otherwise was itself misreading stale data as current.
Annoying, but honest about being wrong once you dug in. The next one wasn't.
The real version: 18,300 tracebacks, and every single check said "fine"
A different bot's scheduled task kept reporting success — exit code 0, every 5-minute run, for over three weeks straight. The scheduler's own logs showed nothing but green.
Inside, the actual Python process had been crashing on nearly every cycle that whole time: an authentication error from the broker's API, unhandled, caught only by the outer process wrapper, which then dutifully reported "the wrapper ran and exited" as success — which was technically true and completely useless. Three weeks of 5-minute cycles is over 8,000 attempts; more than 18,000 tracebacks piled up in the log because a few different code paths kept trying and kept failing. Zero real trades got recorded in that entire window. Nothing about the scheduler's own view of the world ever turned red.
I only found it because I went and read the raw log file directly, not because anything monitoring the system told me to.
Why the tools I already had didn't (and wouldn't have) caught this
The LLM-observability tools I know of trace individual API calls while you're actively building — good for "why did this one prompt cost so much" or "why did this one call return garbage," not built to watch a background job nobody's looking at.
The classic dead-man's-switch tools (the "ping us every N minutes or we alert you" category) would have shown green the entire three weeks, too — the wrapper process itself never stopped running or stopped pinging. That category answers "did the job run." It has no way to know what the job was actually supposed to accomplish, so it can't tell you the job ran and did nothing.
What both incidents have in common: the failure was invisible to anything that only checks "did the process exit 0" or "did something get logged as a plain string." Neither incident involved the code lying — the wrapper genuinely didn't crash, and the "open position" genuinely had been open at some point. The gap was between "the shell of the job looks fine" and "the job actually did the thing it exists to do."
What I'm taking from this
An agent that's competent while you're watching it, and an agent whose failures you'll actually notice once you stop watching, are not the same property — same lesson as the write-access incidents, different failure shape. Uptime monitoring answers "is it alive." Nobody was asking the more useful question: "is it still doing the thing," specifically for a background AI agent where "the thing" is something more structured than "return HTTP 200."
I'm looking at building a small monitoring layer specifically for solo-developer/small-team unattended AI agents — schedule-aware, understands that "the process exited 0" and "the agent did its job" are different claims, and flags the gap between them instead of only the process dying outright.
If you're running any kind of unattended agent — a scraper, a bot, a pipeline — on a schedule with nobody watching, I'd like to know if you've had your own version of "everything said green and it wasn't." Trying to figure out if this generalizes past my own two data points, same as last time.
Top comments (8)
The part I like here is that the task reported on the wrapper, not the work. I try to make scheduled jobs emit one small domain-level proof at the end, even if it is boring. Row count changed, broker state matched, artifact hash moved, that kind of thing. Exit code alone is too cheap to fake by accident.
Mine ran in the opposite direction, which was harder to read: the job reported a hard failure while the write had already landed. The create call succeeded, the step parsing its own response died on a raw control character in one field, and the retry then went to a URL missing the new id and came back 404 — so the log said
API errorwhile the thing had been live on the account the whole time.The outcome-assertion point in this thread is the right one, but the assertion needs a control of its own: I keep one id that must fail the same check (it returns 404 for everyone, including me with my own key), because a checker pointed at the wrong resource passes silently and looks exactly like a healthy run.
The two shapes in the thread so far are "the wrapper reported instead of the work" and, in @vinhnguyenthanhdn's case, the inverse. There is a third one that bit me and it is meaner than both, because the outcome assertion everyone here is recommending does not catch it: the assertion runs fine and reads a stale number.
I collect engagement metrics for a set of published posts on a schedule. One post sat at exactly 323 for two days. Flat is a completely plausible outcome for that metric, so nothing alerted. But the number was not flat, it was last known - the fetch had been failing and the previous value was being carried forward. When the transport came back the real figure was 332, and it had been climbing the entire time.
Two things I would bolt onto the domain-level-proof idea because of it:
Assert freshness per source, not globally. I had a
MAX(collected_at)check and it was green - because one healthy source dragged the max forward while four others were nearly ten hours stale. A global max over sources that can fail independently is a lie the moment they do. Group by source.A missing run has to be loud. In an append-only log, a run that never fired and a run that found nothing are the same row: no row. I now write a record on every cycle including the empty ones, so absence is a visible gap rather than something I have to infer later.
The uncomfortable version of your closing point: 18,300 tracebacks is at least evidence. The failure mode where the number is simply old produces no tracebacks at all, and it passes every check that asks "did the job run" and "did it write something."
Did the raw-log read that found yours turn into a scheduled check, or is it still the thing you do when something feels off?
Yes, twice, and both had the same shape as yours: the layer that reported success was a real layer, it just wasn't the one doing the work.
I capture screenshots of social posts as evidence for a harassment-preservation service. Every capture came back HTTP 200 with a valid PNG. They were Cloudflare block pages. The wrapper I was going through had gone dead, and a block page is still a successful fetch of a real image - the failure arrived wearing the exact shape of the success.
The other one was last week. My publishing script pushes an article to a git-backed platform. Push succeeds, exit 0. The platform has an undocumented posting-rate limit, and over it the article simply never deploys - the URL sits at 404 and nothing anywhere reports an error. I had already misdiagnosed that once as a frontmatter problem, because a successful push felt like evidence.
What I took from it: the check has to assert the thing you actually want, not a proxy for it. Exit 0 means the wrapper returned. 200 means bytes arrived. Neither is "the work happened." Both of mine are now fixed by asserting the outcome instead - the capture is compared against a known-good stencil, and the publish step polls the article URL for a 200 and refuses to record it as published until it sees one. So it does generalise well past two data points.
The useful split here is exit status versus outcome assertion. A wrapper returning 0 is just a claim about the wrapper. For unattended agents, I would want the monitor to check the traded, published, captured, or reconciled artifact directly, even if that check is slower.
On the schedule-aware part, the thing I would build in from the start is each job pushing its own tolerance rather than the monitor holding a schedule per job. Ours emits two values every run: the last-success timestamp, and the max age it is allowed to reach. The monitor then holds two rules in total instead of two per job, and adding a job needs no change on the monitoring side.
One trap that came with it: a window sized for a run that never started is wide, eight days for our weeklies, so a job that ran and went red stayed silent for that long. Failing needs its own signal rather than borrowing the staleness one.
Does your five-minute cadence make the tolerance obvious, or does it vary per bot?
Three weeks of green while it was quietly broken is the nightmare case. The AI summarizer trusting stale logs is really a missing liveness check, success got inferred from "no error" instead of verified output. I started asserting on a real downstream artifact per run, what tipped you off in the end?
Building a monitoring layer focused specifically on unattended background agents hits a massive gap right now. Current LLM observability tools are geared toward prompt debugging and API latency, while Sentry/Datadog just catch hard crashes. Neither tells you if an agent got stuck in a loop or misread stale DB state while returning a green status. If you ship a lightweight tool that lets us set simple outcome-based checks for scheduled background agents, I'd try it out in a second.