Make Escalation Cheaper Than Restart: A Three-Lane On-Call Runbook

If a restart is cheaper to type than a page, your on-call runbook is teaching the wrong habit. I want the first twelve minutes after an alert to prefer observation and escalation over mutation, every time. Why should a language model get a kubectl rollout restart before a human even hears the pager? This piece treats the runbook as a three-lane contract: observe, page, then mutate, with a freeze latch that a human must clear.

I am not claiming a war story from last night's incident, and I am not selling a miracle agent. What follows is a proposed contract plus a small guard you can run locally before any bot is allowed to speak. If the guard cannot classify a command, the command does not ship. That is the whole thesis, and it stays useful even if you never touch an assistant.

The problem the runbook usually hides

Most “AI on-call” drafts collapse three different jobs into one chatty paragraph. The model reads a noisy alert, invents a cause, and then reaches for the most dramatic verb it knows. Restart sounds decisive. Scale-up sounds helpful. Delete-the-pod sounds like surgery. None of those verbs are first commands.

Ask yourself a blunt question: after the alert fires, who owns the next twelve minutes? If the answer is “the bot, unless someone interrupts,” you already lost the lane design. A runbook that lists twenty kubectl spells but never names a pager target is not a runbook. It is a mutation menu with a narrative glued on top.

I keep the lanes boring on purpose. Observe commands may run automatically because they cannot change production. Page commands are the default next step when the observe clock expires. Mutate commands exist, but they sit behind a freeze latch and a human token. Does that feel slow? Good. Slow is the point when the alternative is an invented restart.

Three lanes, one clock, one latch

Here is the contract I actually want checked in beside the service.

• Observe: read-only commands with a hard wall clock, usually three minutes.
• Page: create or raise an incident before any mutating verb is even proposed.
• Mutate: a tiny allowlist that requires an unfrozen latch and a human token.
• Freeze: after any mutate proposal or execution, further mutate traffic is refused until a human unfreezes.

The clock is not decoration. If observation burns three minutes and the service is still red, escalation is cheaper than another theory. Who benefits from a fourth describe pod while the primary on-call is still asleep? Nobody in the blast radius, that is who.

I also refuse mixed argv. A command is observe, page, or mutate. If your wrapper both tails logs and restarts a Deployment, split it. The guard below is deliberately dumb, because clever classifiers are how agents start assuming things.

Artifact: a versioned runbook contract

Label this YAML as a proposal you can edit, not as production law copied from my cluster. Put it in git next to the service so reviewers can argue about verbs, not vibes.

# runbook.payments-api.yaml
# Proposal / example — unexecuted. Review before any real pager or cluster use.
apiVersion: oncall.local/v1
kind: Runbook
metadata:
  service: payments-api
  owner: platform-oncall
  timezone: UTC
clock:
  observeSeconds: 180
  escalateAfterSeconds: 720
  muteRepeatSeconds: 300
alert:
  requiredFields:
    - fingerprint
    - service
    - severity
    - firedAt
    - runbookRef
  failClosedOnMissing: true
lanes:
  observe:
    - id: rollout-status
      argv: ["kubectl", "rollout", "status", "deploy/payments-api", "-n", "prod", "--timeout=30s"]
    - id: recent-events
      argv: ["kubectl", "get", "events", "-n", "prod", "--field-selector", "involvedObject.name=payments-api"]
    - id: error-rate
      argv: ["promtool", "query", "instant", "sum(rate(http_requests_total{service=\"payments-api\",code=~\"5..\"}[5m]))"]
  page:
    - id: page-primary
      argv: ["pd", "incident", "create", "--service", "payments-api", "--urgency", "high"]
    - id: raise-severity
      argv: ["pd", "incident", "update", "--from-alert", "--urgency", "high"]
  mutate:
    - id: restart-deploy
      argv: ["kubectl", "rollout", "restart", "deploy/payments-api", "-n", "prod"]
      requiresHumanToken: true
      requiresUnfrozen: true
      maxBlastRadius: "one Deployment in prod"
freeze:
  latchFile: "./oncall.freeze"
  reasonFile: "./oncall.freeze.reason"
  rule: "After any mutate proposal or execution, refuse further mutate until a human unfreezes."
unfreeze:
  argv: ["./unfreeze.sh"]
  requiresHumanToken: true

Notice what is missing. There is no “ask the model what to do next” field. There is no free-form shell. The alert schema fails closed if fingerprint or runbookRef is absent, because a bot that cannot name the runbook should not pick a lane. That rule is adjacent to incomplete-alert hygiene, but the load-bearing idea here is the clock plus the page lane, not a packet filter.

Artifact: a guard that refuses the wrong lane

The script is a proposal you can run against recorded argv. It does not talk to Kubernetes. It does not page anyone. It classifies, watches the clock, and honors the freeze latch.

# runbook_guard.py
# Proposal / example — unexecuted. Local classifier only; no cluster calls.
from __future__ import annotations

import json, sys, time
from pathlib import Path

import yaml

LATCH = Path("oncall.freeze")
STATE = Path("oncall.state.json")


def load_runbook(path: str) -> dict:
    data = yaml.safe_load(Path(path).read_text())
    if data.get("kind") != "Runbook":
        raise SystemExit("not a Runbook document")
    return data


def lane_for(runbook: dict, argv: list[str]) -> str | None:
    for lane in ("observe", "page", "mutate"):
        for step in runbook["lanes"][lane]:
            if step["argv"] == argv:
                return lane
    return None


def state() -> dict:
    if STATE.exists():
        return json.loads(STATE.read_text())
    return {"alertFiredAt": None, "pagedAt": None, "lastLane": None}


def save(st: dict) -> None:
    STATE.write_text(json.dumps(st, indent=2))


def decide(runbook: dict, argv: list[str], now: float, human_token: bool) -> dict:
    st = state()
    lane = lane_for(runbook, argv)
    if lane is None:
        return {"allow": False, "reason": "argv is not on any sealed lane"}

    fired = st.get("alertFiredAt") or now
    st["alertFiredAt"] = fired
    elapsed = now - fired
    observe_s = runbook["clock"]["observeSeconds"]
    escalate_s = runbook["clock"]["escalateAfterSeconds"]

    if lane == "observe" and elapsed > observe_s and not st.get("pagedAt"):
        return {
            "allow": False,
            "reason": "observe window closed; issue a page lane command first",
            "elapsedSeconds": elapsed,
        }

    if lane == "page":
        st["pagedAt"] = now
        st["lastLane"] = lane
        save(st)
        return {"allow": True, "lane": lane, "elapsedSeconds": elapsed}

    if elapsed >= escalate_s and not st.get("pagedAt") and lane != "page":
        return {
            "allow": False,
            "reason": "twelve-minute clock elapsed without a page; mutate is refused",
        }

    if lane == "mutate":
        if LATCH.exists():
            return {"allow": False, "reason": "freeze latch is set; human must unfreeze"}
        if not human_token:
            return {"allow": False, "reason": "mutate requires a human token"}
        LATCH.write_text("frozen after mutate proposal\n")
        Path("oncall.freeze.reason").write_text("mutate proposed: " + json.dumps(argv))
        st["lastLane"] = lane
        save(st)
        return {"allow": True, "lane": lane, "note": "latch set; further mutate refused"}

    st["lastLane"] = lane
    save(st)
    return {"allow": True, "lane": lane, "elapsedSeconds": elapsed}


def main() -> None:
    if len(sys.argv) < 3:
        raise SystemExit("usage: runbook_guard.py RUNBOOK.yaml -- argv...")
    runbook = load_runbook(sys.argv[1])
    try:
        sep = sys.argv.index("--")
    except ValueError:
        raise SystemExit("missing -- before argv")
    argv = sys.argv[sep + 1 :]
    human = "--human-token" in sys.argv[:sep]
    print(json.dumps(decide(runbook, argv, time.time(), human), indent=2))


if __name__ == "__main__":
    main()

A few commands show the shape. None of these should be aimed at production until you replace the pager binary and the kube context on purpose.

# Proposal / example commands — local classification only.
python runbook_guard.py runbook.payments-api.yaml -- \
  kubectl rollout status deploy/payments-api -n prod --timeout=30s

# After three minutes without a page, the same observe command should fail closed.
python runbook_guard.py runbook.payments-api.yaml -- \
  pd incident create --service payments-api --urgency high

# Mutate without a human token should fail.
python runbook_guard.py runbook.payments-api.yaml -- \
  kubectl rollout restart deploy/payments-api -n prod

# Mutate with a token should succeed once, then freeze.
python runbook_guard.py runbook.payments-api.yaml --human-token -- \
  kubectl rollout restart deploy/payments-api -n prod

# Second mutate should hit the latch.
python runbook_guard.py runbook.payments-api.yaml --human-token -- \
  kubectl rollout restart deploy/payments-api -n prod

The unfreeze path is intentionally ugly. A human writes a reason and deletes the latch. The bot never deletes the latch, even if the model is very polite about it.

# unfreeze.sh — proposal / example
set -euo pipefail
if [[ -z "${HUMAN_TOKEN:-}" ]]; then
  echo "HUMAN_TOKEN required" >&2
  exit 1
fi
echo "unfrozen by human at $(date -u +%FT%TZ)" >> oncall.freeze.reason
rm -f oncall.freeze

Decision table you can paste into the pull request

Would I let a model fill the YAML? Only as a draft. The allowlist is a human artifact because the cost of a wrong verb is paid by whoever is asleep. That is also why escalation sits in front of restart: paging is reversible attention, restart is not.

Where a free assistant actually helps

Drafting this YAML by hand is tedious, and tedium is how observe commands quietly grow a --force. I will use an assistant to propose extra read-only argv, to argue about blast radius wording, and to generate the decision table from the file. I will not give that assistant a kubeconfig.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat MonkeyCode here as a drafting surface: free model access plus a free server option, so the runbook text can be iterated without standing up a GPU box. That is the whole product role in this workflow. No model name, quota, or hardware story is implied, because those details change and I will not invent them.

If you try that path, keep the server on the draft. The guard process, the latch file, and the pager token stay on a machine you already trust. Mixing “helpful autocomplete” with cluster-admin is how assume-happy agents get a restart they were never offered.

A test plan you can run without paging the org

Do not rehearse this against production paging policies on a Friday. Use a fixture alert and a fake pd stub that only writes a JSON line.

1. Feed an alert JSON missing firedAt. The wrapper around the guard should refuse before argv runs.
2. Run three observe commands inside three minutes. All three should allow.
3. Fast-forward alertFiredAt in oncall.state.json by 181 seconds. The next observe command should refuse.
4. Run page-primary. Confirm pagedAt is stamped and the stub file contains one incident.
5. Attempt restart-deploy without --human-token. Expect refuse.
6. Attempt it with --human-token. Expect allow, and oncall.freeze must exist afterward.
7. Attempt a second restart. Expect refuse with the latch reason.
8. Run unfreeze.sh with HUMAN_TOKEN set. Confirm the latch is gone and the reason file kept history.
9. Replay a slightly different argv, such as adding --force. Expect refuse because the list is exact.

If step nine ever starts “helpfully” normalizing flags, you broke the guard. Exact argv is a feature. Fuzzy matching is how restart becomes delete.

Limitations, and who should not use this

This contract does not replace IAM, admission control, or a real incident bot. A local latch file is process-local truth, not a distributed lock. Two replicas of the bot can race unless you put the latch on something both of them honor. The Python classifier also cannot see shell wrappers, so bash -c 'kubectl rollout restart ...' will not match and must stay refused.

Who should skip this approach? Anyone without a real human roster, because a page lane with nobody on the other end is theater. Anyone handing cluster-admin to an unsupervised agent, because the YAML will not save you. Anyone whose “runbook” is still a wiki paragraph with no argv. And anyone who needs sub-minute automated remediation with a signed playbook already reviewed by SRE; this design is slower on purpose.

I also will not pretend the twelve-minute constant is science. Change escalateAfterSeconds when your error budget and your pager load demand it. Just do not let the model edit the clock during an incident. The clock is policy. Policy that mutates itself under stress is not policy.

What I want in the merge request

When this lands, I want four files and one argument. The YAML, the guard, the unfreeze script, and a recorded fixture alert. The argument is simple: did we make paging cheaper than restart, and does the latch survive a second mutate? If reviewers cannot answer those two questions from the diff, the runbook is still a blog post.

So, will your next alert still offer a restart in the first breath? If yes, the lanes are fake. Seal observe, put escalation on a clock, and freeze mutate until a human says otherwise. That is the runbook I will actually keep next to production, and it is the one I am willing to let an assistant draft—never execute.

If you want a quiet place to iterate on the YAML before it hits git, MonkeyCode’s free model access and free server option are enough for drafting. Keep kubectl and the pager token somewhere the draft cannot reach.