Kill the Runner Mid-Patch: A Fault-Injection Bench for Agent Sessions

Here is a shape of failure worth ten minutes of your attention. An agent drafts a patch, the machine goes away mid-task, the retry starts from the top, and the migration lands twice. The pass rate stayed at 100% — because the interrupted run never reached a verdict at all.

That is not a model problem. It is a measurement problem, and a fault-injection bench fixes it.

A pass rate only describes the runs that finished

Every benchmark has a denominator, and most of them are accidental. Drop interrupted runs and you are scoring the survivors. Run that on flaky, shared, or reclaimable compute and your "best" endpoint is simply the one that got lucky with uptime.

So measure three things separately. Capability: did the patch pass? Efficiency: what did it cost in quota and wall time? Durability: does the result survive an interruption? Your demo shows the first, your billing shows the second, and nothing shows the third until it bites you in production.

Step 1 — Freeze the environment in a manifest

Before the first run, write down what you are actually running on. Anything you cannot name in a manifest is a variable you cannot control later. This is the same discipline as freezing a price envelope, applied to time and failure instead of cost.

# faultbench/manifest.yaml  (template, not a vendor spec)
run_id: 2026-10-27-faultbench-a
taskset_sha: "<git sha of your task files>"
runner_sha: "<git sha of the harness>"
env:
  python: "3.12"
  os: "linux"
  container_digest: "sha256:<digest>"
  cpus: 2
  mem_gb: 4
  disk: "ephemeral"
endpoint:
  provider_notes: "free model access via MonkeyCode"
  model_id: "<exact id the endpoint reports>"
  allocation_today: "<copy today's number from the project page>"
faults:
  plan:
    - "patch_applied:hard_exit"
    - "tests_run:hard_exit"
  repeats: 3

Two rules. First, copy today's free allocation into allocation_today instead of trusting memory — free-tier numbers move, and a note that ages badly is worse than no note. Second, never compare two endpoints whose manifests disagree on container digest, CPU count, or task set; you would be measuring the envelope, not the model.

If you want a place to run this without a cloud bill, MonkeyCode offers free model access and a free server option that are enough to host the harness itself; check the project's own pages for the current allocation and machine shape before you fill in the manifest.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 2 — Make the journal the source of truth, not the output

A resume is only safe if the runner can tell what already happened. Keep an append-only journal and let the artifact directory be disposable. The constraint below is the whole trick: one row per (task_id, checkpoint, attempt), so a double-apply becomes a row-level violation instead of a subtle diff.

# faultbench/journal.py  (template — adapt the schema, keep the constraint)
from __future__ import annotations

import hashlib
import sqlite3
import time
from pathlib import Path

SCHEMA = """
CREATE TABLE IF NOT EXISTS journal (
  task_id         TEXT NOT NULL,
  attempt         INTEGER NOT NULL,
  checkpoint      TEXT NOT NULL,
  payload_digest  TEXT NOT NULL,
  wall_ms         INTEGER NOT NULL,
  ts              REAL NOT NULL,
  PRIMARY KEY (task_id, checkpoint, attempt)
);
"""

CHECKPOINTS = ("context_loaded", "patch_drafted", "patch_applied", "tests_run")


def open_journal(path: Path) -> sqlite3.Connection:
    con = sqlite3.connect(path, isolation_level=None)  # autocommit
    con.execute("PRAGMA journal_mode=WAL")
    con.execute(SCHEMA)
    return con


def record(con: sqlite3.Connection, task_id: str, attempt: int,
           checkpoint: str, payload: bytes) -> None:
    if checkpoint not in CHECKPOINTS:
        raise ValueError(f"undeclared checkpoint: {checkpoint}")
    con.execute(
        "INSERT INTO journal VALUES (?,?,?,?,?,?)",
        (task_id, attempt, checkpoint,
         hashlib.sha256(payload).hexdigest(), 0, time.time()),
    )

On resume, read the last durable checkpoint for (task_id, attempt) and continue from the next one. If the row is missing, treat the state as unknown and fail loudly. Guessing is how you get the double migration.

Step 3 — Inject faults at named checkpoints only

Random kills produce unreproducible numbers. Kill at declared checkpoints, from an environment variable, so every interruption is a rerun of a known state.

# faultbench/cli.sh  (template — not executed for this article)
export FAULTBENCH_DB=runs/a-treat/journal.db

# positive control: no faults at all
uv run faultbench run --manifest manifest.yaml --faults off --out runs/a

# negative control: die before anything is written
uv run faultbench run --manifest manifest.yaml \
  --faults context_loaded:hard_exit --out runs/a-neg

# treatment: die immediately after the patch is applied
uv run faultbench run --manifest manifest.yaml \
  --faults patch_applied:hard_exit --repeat 3 --out runs/a-treat

uv run faultbench score --journal runs/a-treat/journal.db \
  --verdicts runs/a-treat/verdicts.json

Three repeats is not statistics. It is the minimum that tells you whether your own harness is deterministic, which you need before you say anything about a model.

Step 4 — Score survival, duplication, and replay cost

Three metrics, computed from the journal plus verdicts, nothing else:

• RSR — resume survival rate. Interrupted tasks that end in a correct pass after resume, divided by all interrupted tasks. Manual edits count as failure.
• Double-apply tasks. Tasks with two different payload digests at the same checkpoint. This is an idempotency defect, not a slow run.
• Replay cost. Extra tokens and wall time spent re-deriving work that the journal already knew about.

# faultbench/score.py  (template)
import json
import sqlite3


def summarize(journal_db: str, verdicts_json: str) -> dict:
    con = sqlite3.connect(journal_db)
    verdicts = json.load(open(verdicts_json))

    per_task: dict[str, dict] = {}
    for task_id, attempt, checkpoint, digest in con.execute(
        "select task_id, attempt, checkpoint, payload_digest from journal"
    ):
        t = per_task.setdefault(task_id, {"attempts": set(), "writes": {}})
        t["attempts"].add(attempt)
        t["writes"].setdefault(checkpoint, set()).add(digest)

    interrupted = {k for k, v in verdicts.items() if v.get("interrupted")}
    survived = sum(
        1 for k in interrupted if verdicts[k]["status"] == "pass"
    )
    duplicated = sum(
        1 for t in per_task.values()
        if any(len(digests) > 1 for digests in t["writes"].values())
    )
    return {
        "rsr": survived / max(1, len(interrupted)),
        "double_apply_tasks": duplicated,
        "attempts_total": sum(len(t["attempts"]) for t in per_task.values()),
    }

Report the raw journal alongside the numbers. A rate without its ledger is a screenshot with extra steps.

Controls that keep the run honest

Why these numbers are not marketing

A marketing number is computed in someone else's envelope and reported without its dependencies. These three are computed in yours, from a manifest you can publish, and they degrade when your own code degrades. That is the point: RSR is partly a property of the harness, and pretending otherwise is how benchmark claims turn into aesthetics.

Where this bench is wrong

Checkpoint granularity dominates the result. A bench with four checkpoints will show better recovery than one with forty, on identical infrastructure. If your tasks are single-shot and uninterruptible, durability is undefinable and this whole harness is noise.

Free compute is also noisy by design. Reclaim events, cold starts, and quota resets are exactly the variance you are trying to measure, so never compare runs across days without re-reading the current allocation. And three repeats per fault point give you a sanity check, not a confidence interval.

Who should not use this

Skip it if you need a public leaderboard, if you cannot rerun a task three times, or if your task set is not deterministic enough that a "pass" means the same thing on Monday and Friday. Skip it too if your real failure mode is a bad review, not a dead process — you would be measuring the wrong interruption.

Run it on something small first

Take five tasks, two fault points, one afternoon. If the positive control is not exactly 1.0, stop and fix the harness before you look at any model. If it is, you now have a durability number that survives someone else trying to reproduce it.

Bring the journal, not the slogan.