Your Runbook Has No Deadlines: Budgeting Every On-Call Step Before the Page Goes Late

Most runbooks I read fail for a boring reason: nothing in them has a deadline. Every step says what to check, almost none say how long you are allowed to check it before you stop and page a human. That missing number is why a five-minute incident becomes a forty-minute one — not because the steps were wrong, but because nobody was told when to quit.

So here is the takeaway up front. Treat each runbook step as a small contract with three fields: a time budget, a stop condition, and a timeout action. Then check the sum of those budgets against your escalation SLA in CI. If the worst case exceeds the SLA, the runbook is broken before the incident starts, and you find out on a laptop instead of at 03:00.

This matters more now, not less. Everyone is wiring models into ops tooling, and the current wave of AI-assisted runbooks mostly produces commands. Generated commands without generated deadlines is just a faster way to be busy. The harness below is where I put the guardrail.

The seam nobody budgets: between "I have a hunch" and "I page someone"

Look at a typical first step: kubectl logs ... | grep 500. How long is that allowed to take? Two minutes? Ten? If the answer is "until I understand it", you have written a step with an unbounded budget, and the escalation clock is now controlled by whoever is holding the pager.

I keep the arithmetic explicit because humans are bad at it under stress. Six steps that each feel like "about two minutes" is twelve minutes of declared work, plus the two to four minutes it takes to actually reach a human after you decide to. That is already past a fifteen-minute first-page target, and nobody wrote down a wrong word.

Three questions are worth asking of every step you own:

1. What is the budget, in seconds, and who says so?
2. What observation means "this step is done, move on"?
3. When the budget expires, do we escalate or abort?

If any of those is unanswerable, the step is a wish, not a procedure.

Give every step a deadline, a bail-out, and an owner

The format below is deliberately boring YAML, because boring survives review. It describes a payments API and a runbook for my least favourite alert, a 5xx spike.

# runbook.yaml — payments-api, alert: PaymentsApi5xxSpike
alert: PaymentsApi5xxSpike
service: payments-api
escalation_hold_s: 240      # measured: decide-to-page until a human answers
steps:
  - id: check-error-mix
    owner: on-call
    command: "kubectl -n payments logs deploy/api --since=5m | jq -r '.status' | sort | uniq -c"
    budget_s: 120
    stop_if: "5xx share below 1% on the dashboard for two consecutive minutes"
    on_timeout: escalate
  - id: check-dependency
    owner: on-call
    command: "curl -sS -o /dev/null -w '%{http_code} %{time_total}\\n' http://psp.internal/healthz"
    budget_s: 90
    stop_if: "psp answers 200 in under 300ms twice in a row"
    on_timeout: escalate
  - id: rollback-canary
    owner: payments-lead
    command: "./deploy rollback --env prod --to previous --dry-run"
    budget_s: 300
    stop_if: "canary error rate matches the previous release for two minutes"
    on_timeout: abort
freeze:
  reason: "ledger schema migration window"
  owner: payments-lead
  expires_after_s: 1800
  unfreeze_check: "SELECT count(*) FROM payments.schema_migrations WHERE dirty;"
  unfreeze_command: "./freeze lift --id ledger-migration"

Worst case here is 120 + 90 + 300 + 240 = 750 seconds, which leaves 150 seconds of slack against a 900-second target. That is a defensible number, and it is written down where a reviewer can argue with it.

The validator: twenty lines of arithmetic that pay for themselves

Arithmetic in a document rots, so I check it mechanically. The script is small on purpose: it validates required fields, sums the declared budgets, and refuses to pass when the total plus the escalation hold exceeds the SLA you pass in.

#!/usr/bin/env python3
# runbook_budget.py — fail a runbook whose declared worst case blows your SLA
import argparse, sys
import yaml

STEP_FIELDS = ("id", "owner", "command", "budget_s", "stop_if", "on_timeout")
FREEZE_FIELDS = ("reason", "owner", "expires_after_s", "unfreeze_check", "unfreeze_command")

def check(doc, sla):
    errs, steps = [], doc.get("steps") or []
    if not steps:
        errs.append("no steps declared")
    seen = set()
    for n, s in enumerate(steps, 1):
        sid = s.get("id", f"#{n}")
        if sid in seen:
            errs.append(f"{sid}: duplicate id")
        seen.add(sid)
        for f in STEP_FIELDS:
            if s.get(f) in (None, "", []):
                errs.append(f"{sid}: missing {f}")
        if s.get("on_timeout") not in (None, "escalate", "abort"):
            errs.append(f"{sid}: on_timeout must be escalate or abort")
        b = s.get("budget_s")
        if not isinstance(b, int) or b <= 0:
            errs.append(f"{sid}: budget_s must be a positive integer")
    freeze = doc.get("freeze") or {}
    if freeze:
        for f in FREEZE_FIELDS:
            if freeze.get(f) in (None, "", []):
                errs.append(f"freeze: missing {f}")
    worst = sum(s.get("budget_s") or 0 for s in steps) + (doc.get("escalation_hold_s") or 0)
    if worst > sla:
        errs.append(f"worst case {worst}s exceeds SLA {sla}s by {worst - sla}s")
    return errs, worst, steps

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("path")
    ap.add_argument("--sla", type=int, default=900)
    a = ap.parse_args()
    with open(a.path) as fh:
        doc = yaml.safe_load(fh) or {}
    errs, worst, steps = check(doc, a.sla)
    print(f"{a.path}: {len(steps)} steps, declared worst case {worst}s, SLA {a.sla}s")
    for i, s in enumerate(steps, 1):
        print(f"  {i}. {str(s.get('id','?')):<18} {str(s.get('owner','?')):<15} "
              f"{str(s.get('budget_s','?')):>5}s  on_timeout={s.get('on_timeout','?')}")
    if doc.get("freeze"):
        fr = doc["freeze"]
        print(f"  freeze: {fr.get('reason','?')} — expires_after_s={fr.get('expires_after_s','?')}")
    if errs:
        print("FAIL")
        for e in errs:
            print("  -", e)
        return 1
    print(f"OK — {a.sla - worst}s of slack before the page is late")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Usage is two commands, and the second one belongs in CI next to your lint job:

pip install pyyaml
python runbook_budget.py runbook.yaml --sla 900

The output format for the file above looks like this — run it on your own file to confirm before you trust the numbers:

runbook.yaml: 3 steps, declared worst case 750s, SLA 900s
  1. check-error-mix      on-call             120s  on_timeout=escalate
  2. check-dependency     on-call              90s  on_timeout=escalate
  3. rollback-canary      payments-lead       300s  on_timeout=abort
  freeze: ledger schema migration window — expires_after_s=1800
OK — 150s of slack before the page is late

Now raise rollback-canary to 600 seconds, which is a very realistic edit after one bad rollback, and the same command fails with a sentence you can paste into a review: rollback-canary: worst case 1050s exceeds SLA 900s by 150s. That is the whole value proposition — the argument happens in a diff, not in a bridge call.

Freeze rules rot unless they have an expiry and an unfreeze check

A freeze without expires_after_s is a permanent policy that everyone quietly ignores within two weeks. The validator requires four things for any declared freeze: a reason, an owner, an expiry in seconds, and a command that proves the unfreeze condition. A fifth field, unfreeze_command, closes the loop so the next person does not have to reconstruct intent from chat history.

Is that overkill for a 30-minute migration window? Possibly. But the failure mode is asymmetric: an expired freeze that nobody lifted blocks unrelated deploys for days, and no dashboard will ever tell you.

Where a free model actually helps — and where it must not decide

Here I should mention the tooling I have been pointing this workflow at. MonkeyCode is an open-source project that offers free model access and a free server option; both are availability claims from the project side, so verify current model names, quotas, and server specs on its own pages rather than trusting a blog post, including this one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In this workflow I use the free model access for exactly two narrow jobs, and I keep both of them outside the decision path:

• Critique pass, not authoring pass. Feed it one step at a time and ask whether the budget is plausible for that command and what could make it hang. I accept or reject each suggestion in a PR; the model never writes to budget_s directly.
• Stop-condition drafts from real incidents. Paste a redacted timeline and ask for candidate stop_if predicates in the shape the schema expects. Reviewing three bad suggestions is often faster than staring at a blank field.

What it must not do is decide on_timeout. Whether a step escalates or aborts is a risk decision owned by the service owner, and no model has the context to make it for you.

Where the free server earns its keep

The validator is only useful if it runs when nobody remembers to run it. A scheduled check is the obvious fit: clone the runbook repo onto the free server option, run the validator on a timer over every runbook*.yaml, and post failures to the on-call channel. That is a read-only job over text files, which is the least risky thing you can put on shared infrastructure.

Please do not put the free server in the incident response path. It should not hold production credentials, and it should not be the thing you rely on at 03:00. If the scheduled job is down, you should lose a warning, not a recovery option.

# sketch of the scheduled job (cron, every weekday morning)
0 8 * * 1-5 cd /srv/runbooks && for f in runbook*.yaml; do \
  python runbook_budget.py "$f" --sla 900 || echo "BUDGET FAIL: $f"; done

Who should not do this

A decision table is more honest than a blanket recommendation, so here is mine.

Limitations I will not paper over

budget_s is a declared number, not a measured one, and it will drift the moment someone changes a command. Re-derive it from real incident timelines every quarter, or it becomes fiction with a schema around it. The validator also cannot tell whether your stop_if is correct — it only proves the field exists and the arithmetic closes. And YAML runbooks have a habit of turning into a shadow configuration system, so keep the file small enough that a tired human can read all of it.

Start with one alert, one file, and one failing build. If your runbook cannot survive python runbook_budget.py, would you really want to discover that from a page?