In April our card payments failed for an hour and forty minutes before anyone was paged. The alert for it had fired at six minutes past two. In Alertmanager it was active from that moment, and the notification reached the payments team at eight minutes to four.
Alertmanager groups alerts before it tells anyone. Our route grouped by team, so everything for payments arrived as one notification, and three timers controlled it. group_wait is how long a new group waits before its first notification. group_interval is how long to wait before notifying about new alerts added to a group that has already notified. repeat_interval is how long before an unchanged notification is sent again.
In January a disk warning on an archive server had been firing for days and paging the payments channel every few hours, and somebody set group_interval to two hours on that route to quiet it. repeat_interval was the timer that sent those reminders, but the change seemed to work, and nobody looked again. The disk warning went on firing for the next three months, which kept the payments group alive the whole time. When the payment errors alert fired, it joined a group that had already notified, so Alertmanager held it until the next group interval, and the next one was a hundred and six minutes away.
Alertmanager did what the configuration said, and the configuration said that anything new for payments could wait two hours as long as something old was still firing.
We now group by alertname and service, so an unrelated warning cannot hold a group open for anything else. group_interval is back to five minutes on every route, and reminders are controlled with repeat_interval, which is the timer meant for them. Pages go through their own route with short timers and no grouping across services. Routing changes are tested with amtool against a set of example alerts, in the same pipeline that applies them. And once a day a synthetic alert fires through the whole chain, and we record the time from firing to the message arriving, which is the number we should have been watching all along. The archive disk was fixed too, after three months in which the warning had become furniture.
Whether a notification is late depends on what else is already in its group. We had tuned one alert and changed the delay for all of them.
– Sergey Shinder
Top comments (0)