The outage lasted eleven minutes, which everyone was pleased about, and we never found out what caused it. Those two facts are related, and it took me a long time to admit that the second was the price of the first.
What happened was ordinary. A core service started refusing connections, the on-call engineer did what our runbook told him to do, which was restart the service, and the restart worked. Traffic recovered. The incident was closed as resolved within the hour. Three weeks later it happened again, and again the restart fixed it, and again we had nothing to look at afterwards because the restart had taken the process memory, the open file handles, the thread state and the in-flight connections with it.
We were not careless. We had logs. But logs only contain what somebody previously decided to write down, and nobody had previously decided to write down whatever this was. The answer was almost certainly sitting in the state of that process, and we threw it away twice in a row because throwing it away made the graphs go green, and green is what we were measured on.
The change we made was small and slightly uncomfortable. Before any restart during an unexplained incident, capture first: a thread dump, a heap dump if it is feasible, the connection table, the output of the few diagnostic commands we know we will want later. We wrote it as a script so it takes about forty seconds and nobody has to remember the flags under pressure. Forty seconds of extra downtime, once, in exchange for the possibility of not being here again next month.
The harder part was giving the on-call engineer explicit permission to spend those forty seconds. People restart instantly because they believe, correctly, that the organisation rewards fast recovery and is indifferent to understanding. If you want diagnosis, you have to say out loud that a slightly longer outage with evidence is a better outcome than a shorter one without it, and you have to mean it when the report goes upstairs.
A recurring incident you can reproduce is a bug. A recurring incident you cannot reproduce is a permanent resident.
– Serguey Shinder
Top comments (0)