Our checkout service has a latency alert: page someone if the ninety ninth percentile stays above two seconds for ten minutes. In the spring we rewrote it. The old version averaged a percentile across pods, which we had learned is not a percentile of anything, and the new one followed the documentation's advice to add up the histogram buckets first and take the quantile afterwards. It was reviewed, merged and loaded, and Prometheus listed the rule as healthy.
In September a slow query in the basket service pushed checkout's p99 to six seconds for most of an hour. Customers noticed. The latency panel on the dashboard showed it plainly. Nobody was paged.
The rule read:
histogram_quantile(0.99, sum by (service) (rate(http_request_duration_seconds_bucket[5m])))
A Prometheus histogram is a set of counters, one per bucket, told apart by the le label, which holds each bucket's upper bound. histogram_quantile needs that label to know which bucket is which. sum by (service) keeps only the service label, so it added every bucket together into one series and the le label was gone before histogram_quantile saw the data. The query does not fail when that happens. It returns nothing, and an alert whose expression returns nothing is an alert that is not firing. The dashboard panel had been written separately, with sum by (le), which is why it could see what the alert could not.
For five months the rule had been evaluated every thirty seconds and produced no result each time. The rules page showed it as healthy throughout, because health only means the query ran without an error.
The fix was one word: sum by (service, le). Then we added tests. Every alerting rule now has a promtool test file with synthetic series, including at least one case in which the alert must fire, and CI runs promtool test rules on every change. A nightly job also takes each alert's expression, removes the final comparison, runs what is left against production and reports any expression that returns no series at all to the team that owns it. On the first night it found two more, one of them with a misspelt metric name.
An alert that cannot fire looks exactly like an alert with nothing to say. We now make every rule show us, once, that it can fire.
– Sergey Shinder
Top comments (0)