DEV Community

ilya mozerov
ilya mozerov

Posted on

The sensor was fine. My safety gate was the thing that went blind.

A monitoring tool I run at home has one job: write the current posture of the machine into a
state file, every five minutes, so that anything downstream that asks "how are we doing?" gets
an honest answer.

For 2026-09-15 through 2026-09-21 it didn't. And the tool never errored — it did its
safety check, correctly, on every single run, and refused to write.

What that looks like

The state file held 2026-09-15's posture through 2026-09-21. Not because the sensors were
down. Internal health read fine. External conditions read fine. The tool knew, on every run,
that nothing was wrong — and it wrote nothing.

The cause was in the guard, not the measurement. Before committing a verdict, the tool checked
that all twenty of its producers were reachable. Any one of them dark — one flapping Wi-Fi
statistic, one NIC counter that didn't render — and the gate would not open. It emitted BLIND
and moved on, 96 times a day, and the file went stale.

Read that gate as an intention and it is unimpeachable: don't publish a verdict you can't
fully back
. Read it as behavior and it is the exact failure a safety gate exists to prevent.
Downstream was now reading yesterday's posture with a tool that had certified itself
unwilling to speak. A silent system is worse than a wrong one, because a wrong one gets
questioned.

The fix was not to lower the standard

The tempting move is to widen the gate, accept the staleness, publish anyway. That loses the
protection. The actual fix separates two questions the gate had fused:

  • What is the verdict? Carried by two core axes only — internal and external.
  • What could I see? Published alongside it.

Now the edge evaluates unless a core axis is dark. If nineteen of the auxiliary producers are
unreachable, the verdict still commits, and the blindness ships inside it: an axes_unseen
list, plus a (partial — N/4) suffix on the posture line. A machine consumer can now do what
it could not before: refuse to trust a low-coverage NOMINAL, and know precisely which eye was
closed when it was drawn.

The principle, which I have come to believe generalizes past my sensor graph:

A readiness gate must never be stronger than the verdict it guards.

A gate calibrated to demand total coverage before a partial answer produces silence by
construction. Making it weaker is not the lesson. The lesson is that coverage is itself
publishable
, and that "NOMINAL has to mean calm and I could see" is one claim with two
halves — neither of which may be dropped.

Why I think this isn't just my code

Every monitoring system I have built or maintained eventually grows a readiness check, and the
check almost always accretes conditions: wait for all exporters, wait for the baseline window,
wait for every member of the fleet. Each addition is defensible. Together they invert the
system's purpose. The gate becomes the thing that goes blind, and because it fails by not
writing instead of by writing wrong, it is invisible in the dashboards that count errors.

The diagnostic I now use: ask what the gate does when it fails. If the answer is "nothing," the
gate is a silence generator. If the answer is "publishes a bounded answer with the blindness
named," it is a gate.

What I measured

Ran on the fixed tool, before writing this:

scripts/mesh-situation --test      exit 0
scripts/mesh-situation --json      posture=WATCH, internal=WATCH, external=NOMINAL,
                                   axes_unseen present, UNKNOWNs preserved
scripts/mesh-situation --edge      exit 0
.situation.state                   "WATCH", mtime 1 minute ago — actively written,
                                   the stale-window era is over
Enter fullscreen mode Exit fullscreen mode

Every line of the render that is not a core verdict carries its own provenance or its own
remediation. An unreachable producer renders UNKNOWN with the command that would fix it, not
a fallback to a default. There is no path in the tool that prints a calm label it cannot see.

What I did not verify

  • The freeze window (2026-09-15 → 2026-09-21) is the change author's record, written in a source comment next to the fix. I did not personally archive the original logs, and I have no independent timeline to add to theirs. My verification covers the post-fix tool, not the historical incident.
  • I did not replay the failure. I read the mechanism, understood why it would produce exactly this failure, and verified the post-fix behavior above. The failure is a consequence of the old code, not a claim I re-derived at runtime.
  • The counts move. At verification time the render carried 4 axes unseen against 4 core coverage counters — partial coverage is live and being counted, which is the point. That number is a live reading, not a constant; it will differ when you run it.

The shape of it

If you are building something similar, the whole lesson is one line: when the observer cannot
see, publish the blindness with the verdict. Refusing to answer is not caution. It is a
silent all-clear, signed by the safety check.

Top comments (0)