Our overnight processing chain was scheduled to start at ten in the evening and to be finished by five. It had been designed around those numbers in 2014 and had grown a little every year since, in the way these things do, never enough for anyone to notice on the day. One Tuesday it finished at seven forty in the morning, and nothing alerted, because nothing had failed.
Two things then went wrong, and only one of them was visible. The visible one was that the morning reports were not there when people arrived. The invisible one was worse. Half of the jobs in the chain were scheduled by clock time rather than by dependency, an arrangement that had worked for a decade because the durations were comfortable. That night one job started at its appointed hour while the job it depended on was still running, read the previous day's data, and produced a set of figures that were internally consistent, plausible and wrong. Nobody questioned them, because they looked exactly like every other day's figures.
The third thing was waiting for us the following night, when the next run began while the previous one was still writing, and two instances of the same job started updating the same tables.
None of this was a hard engineering problem. Jobs now wait for their predecessors rather than for the clock. A run cannot start while the previous run is active. Runtime is recorded per job and trended, with an alert when total elapsed time crosses eighty percent of the window rather than when something breaks. And we wrote down the actual commitment, which is that the data must be usable by seven, and we measure against that instead of against a list of exit codes.
What I keep coming back to is that every individual job had an owner and the chain had none. Each team could truthfully say their part ran fine. The batch window was a promise about time that no monitoring anywhere was measuring, and gradual growth never trips a threshold that nobody set.
We were alerting on jobs failing. The thing that mattered was jobs finishing late.
– Serguey Shinder
Top comments (0)