A customer emailed about failing uploads. I opened the service dashboard expecting a spike and found a clean, calm, flat line: error rate zero, throughput steady, latency unchanged, all the way across the last four hours. The dashboard was not telling me the service was healthy. It was telling me nothing at all, and it looked exactly the same either way.
The metrics exporter had been killed. It ran as a sidecar with a 64 mebibyte limit, something grew, the OOM killer took it, and the main container kept serving traffic so the pod stayed Ready. Scrapes began failing. Prometheus stopped receiving series. Grafana, with "connect null values" set on every panel because someone thought gaps looked untidy, drew a straight line from the last known point to the right edge of the graph.
The alerts did not fire, and that is the part worth understanding. Our rule was rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.02. With no series in the range, that expression returns no data. Not false. No data. An alerting rule that evaluates to an empty vector produces zero alerts, so the absence of metrics is indistinguishable from perfect behaviour. Every ratio alert we owned had this property. We had built a system where the monitoring dying was the quietest possible state.
What we added was alerts on absence rather than on values. up{job="api"} == 0 for two minutes. absent_over_time(http_requests_total{service="uploads"}[10m]) for anything whose traffic should never legitimately reach zero. A scrape-staleness check on the timestamp of the newest sample per target. And a watchdog rule that always fires into an external heartbeat service, so if our own Alertmanager stops delivering, something outside the cluster notices within five minutes.
Then some unglamorous hygiene: null values render as gaps, not lines; panels show "No data" rather than an empty axis; and the exporter's memory limit was raised after we actually measured it.
The failure I had never planned for is the monitoring system failing silently while the thing it watches keeps running. Flat is not a value. Treat no data as an alertable state, or your dashboards will be most reassuring exactly when they are blind.
– Sergey Shinder
Top comments (0)