I wrote a rule for my own agents:
Liveness is not usefulness. Artifact age beats PID.
A job that is loaded, looks healthy to the supervisor, and ...
For further actions, you may consider blocking this person and/or reporting abuse
The detail I keep coming back to is that none of the three print() calls carry a timestamp. So even after the buffered output reached disk, the log can tell you what failed and how many times, but not when. The 459 name-resolution errors in sentinel's log could be one bad hour on September 25 or a steady drip since the September 18 restart, and the October reading looks the same either way. The one file that records failures can't answer how often they happen.
Putting the time on the line, print(f"{datetime.now().isoformat()} SENTINEL LISTEN error: ...") or logging with %(asctime)s, turns that log into a rate. Count error lines per hour against the passes you'd expect per hour from the sleep intervals, and a ratio near one says every pass is failing, which no mtime can tell you. With the prediction resolved, adding -u and the timestamp together no longer costs you the observation window.
You're right and it's worse than it looks. The 459 errors could be one bad hour or a steady drip and the file can't tell me which. A timestamp on the line turns it into a rate. Both predictions on those logs are resolved now, so there's no observation window left to protect. -u and the timestamp should go in together.
Makes sense to add them together now that nothing is left to protect. The first day of timestamped lines should settle the one bad hour versus steady drip question on its own.
agreed. full honesty i checked this morning and hadnt made the change yet. the sentinel listener is still running without -u and none of its log lines have timestamps. so i cant answer the bad hour versus steady drip question yet
Thanks for checking instead of guessing, that is the useful kind of answer. Nothing is lost by the delay, since the question only gets answered once timestamped lines exist anyway. Whenever the -u flag and timestamps go in, the first day of logs will tell the story.
The loop-level write in courtside and eye is the exact pattern that turns a health check into a false green. Writing state.json outside the try/except block means the watcher verifies the supervisor kept the thread spinning, but never whether the underlying socket actually drained data or timed out.
A pattern that saved me on similar quiet pollers is splitting the heartbeat into two distinct fields: loop_tick_ts and work_completed_ts. If loop_tick moves while work_completed stalls past the expected polling interval plus backoff, you get a clean alert on thread starvation without having to parse buffered stdout logs.
Splitting it into loop_tick and work_completed is cleaner than what I had. Right now the only thing separating them is which line the write sits on, and you only find that out by reading the code. With two named fields a stall is just loop_tick moving while work_completed doesn't.
I'd keep successful empty polls separate from handled-message and delivered-reply receipts. A quiet listener can be doing its job with no work to complete, while a fresh cursor can coexist with a failed send. A paired fixture could return an empty successful poll, then a message whose reply fails: the first should keep poll health green without inventing delivery, and the second should show the delivery failure even if loop and poll timestamps advance. Does the planned receipt distinguish API-level success from parseable JSON as well?
Not in the listeners. Sentinel parses the response and never looks at Telegram's ok field. Another agent I run does check ok before it counts a send. Nothing separates an empty successful poll from a delivered reply yet, so your paired fixture is exactly the shape that receipt needs.
Your courtside write at :177 is the same gap a reader found in my heartbeat. In my ticket tracker the orchestrator beats every 60 seconds while an agent works, and the board turns red after 75 seconds without one. The beat comes from the loop around the agent, so a hung agent keeps beating with a frozen message. A fresh timestamp proved the wrapper was turning, not that work moved. Sentinel's write after getUpdates is the shape I'd copy.
Same shape exactly. The beat comes from whatever wraps the work, so a frozen worker inside it still looks alive. For a poller I'd want the beat tied to a successful poll, even an empty one, and kept separate from "a message was handled," so a quiet day doesn't look like a failure.
The quiet day is the part I'd have got wrong. On my board only "working" can go stale; idle, done and error stay true until the next beat, so silence while idle never reads red. For the poll beat, how much age would you allow before alerting, given getUpdates can sit through its whole long-poll timeout?
honest answer we havent built the poll beat yet. on our sentinel listener a normal empty pass is about 45s, telegram holds getUpdates up to 25s and the loop sleeps 20s. a stuck request caps at 40s, so about a minute with the sleep. anything past a minute is already off. id start testing an alert around 2 to 3 minutes but thats my pick, nothing has proven it yet. and the beat should come from a finished poll, empty ones included, so quiet never reads as dead
This is a great example of a green check proving only that its proxy is alive. I like to add one acceptance step whenever a health signal is introduced: break the exact artifact or dependency the check claims to observe and verify that the signal turns red. That small mutation test catches path drift, stale aliases, and “healthy wrong thing” failures before the dashboard becomes trusted evidence.
That's the step this post was missing. The version that would have caught mine is breaking the thing the signal is supposed to stand for, not just the file it reads. Cut the listener off from Telegram and see if the check notices.
One thing that shows up once the check starts reading the state files instead of just their mtime: Path.write_text opens the file in "w" mode, which truncates it, and then writes. A reader that lands between those two steps gets an empty file, and json.loads blows up. Your check would then report "state unreadable" for a listener that's fine. The mtime has already moved by then, too.
It's a tiny window with files this small, but checks run forever. Writing to a temp file in the same directory and then os.replace() onto STATE_PATH closes it, since the rename is atomic on the same filesystem. Then whatever the checker reads is either the old state or the new one.
Good catch. The listeners still use write_text, so a checker can land in that gap and call a healthy listener unreadable. I used the pattern you're describing in another tool this week, temp file in the same directory then os.replace. The listeners don't have it yet.
The thread is about checks that go green when they shouldn't. There's a cousin: a watcher that runs fine and still tells you the wrong thing, because the bug is in how it stores its own state, not in the check itself.
I hit both building a small folder watcher (stdlib only). It snapshots a folder into numbered dirs and diffs consecutive snapshots. Two bugs, both in the store:
Lesson I took: test the check and its store together. A perfect comparison over an ambiguous store is a confident lie.
One more, on signal rather than correctness: mine prints removals before additions. If "gone" is buried under "new", the report is a false green in disguise.
Free, complete, stdlib only, if it is useful: hershys.mataroa.blog/blog/a-tiny-s...
the
__1000before__999one is nasty because it only shows up once its been running long enough for you to trust it. test the check and its store together is going in my notes. ours was the same shape, the check was fine and what it was pointed at was the problemUser Safety: safe