48-Hour Field Notes: My Overnight Batch Forecast Used the Mean and Ignored the Restart Tax

Have you ever sized an overnight batch job on paper, watched it finish comfortably inside the estimate, and then found it still churning at breakfast? I have, twice this month, and both times the arithmetic was correct and the assumption underneath it was wrong. This is a 48-hour field note about the difference between an average and a promise, and about the hidden cost of every restart on an interruptible box.

The setup was ordinary: a few thousand items, one model call per item, a checkpoint file so nothing gets done twice, and a machine that can reboot under me. I ran it on MonkeyCode's free model access and its free server option, which is what made the restart question unavoidable rather than theoretical. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Hour 0–12: the pilot I should have run first

My first version of the forecast was one line: item count times average call time. It looked responsible because I had measured something instead of guessing, and the measurement came from a sixty-item sample I had actually run. What I never asked was whether that sample resembled the tail of the distribution, or whether the box would stay up for the whole window.

Two things broke that forecast, and neither of them was the model being slow on average.

1. The distribution has a tail. Most calls returned quickly, a few took several times longer, and the queue inherited the tail because it processes every item once.
2. The checkpoint granularity set the restart tax. Saving every tenth item meant a reboot could replay up to nine finished calls, and I had counted that as zero.

Neither problem shows up in a mean. Both show up in a p90 and in a replay counter.

Hour 12–36: measuring instead of assuming

The harness below is the shape I now use before committing to any overnight run. Treat the printed numbers in the next section as illustrative output, not as a benchmark of any provider.

# pilot.py — run this on the same box that will run the batch
import random, statistics, time, json


def pct(values, p):
    s = sorted(values)
    if not s:
        return float("nan")
    k = min(len(s) - 1, max(0, round((p / 100) * (len(s) - 1))))
    return s[k]


def pilot(items, call, sample=60, seed=7):
    rng = random.Random(seed)
    picks = rng.sample(items, min(sample, len(items)))
    rows = []
    for i, item in enumerate(picks):
        t0 = time.monotonic()
        try:
            out = call(item)
            ok = out is not None
        except Exception as exc:          # record, do not swallow
            out, ok = repr(exc), False
        rows.append({
            "i": i,
            "seconds": round(time.monotonic() - t0, 3),
            "ok": ok,
        })
    return rows

Then the forecast itself, which is deliberately pessimistic in two places. It prices items at p90 rather than the mean, and it multiplies by a restart tax you measured rather than one you imagined.

# forecast.py — pessimistic on purpose
def forecast(n_items, rows, p=90, checkpoint_every=1, restarts=1):
    ok = [r["seconds"] for r in rows if r["ok"]]
    failures = len(rows) - len(ok)
    per_item = pct(ok, p)
    # worst case: each restart discards up to checkpoint_every-1 finished items
    replay = restarts * (checkpoint_every - 1)
    billed_items = n_items * (1 + failures / max(len(rows), 1)) + replay
    total_s = billed_items * per_item
    return {
        "p50_item_s": pct(ok, 50),
        "p90_item_s": pct(ok, p),
        "sample_failures": failures,
        "billed_items": round(billed_items, 1),
        "eta_hours": round(total_s / 3600, 2),
    }

If the free server can be restarted by something outside your control, then restarts is not a what-if parameter, it is a budget line. I set it from observed behaviour over the pilot window and then doubled the checkpoint interval's worst case on purpose.

Measuring the restart tax with a kill, not a guess

You cannot estimate replay cost by reading your checkpoint code, because the real number depends on when the interrupt lands. So I interrupt it on purpose, twice, at known item counts.

# terminal 1
python run_batch.py --limit 400 --checkpoint ./ckpt.jsonl

# terminal 2, after ~90 seconds
kill -TERM $(pgrep -f run_batch.py)
wc -l ./ckpt.jsonl            # finished items at the moment of death
python run_batch.py --resume ./ckpt.jsonl --limit 400

The difference between the checkpoint line count and the item the worker was holding is your replay window in practice. Divide replayed items by total items and you have a tax rate you can put into the forecast instead of a hope.

Hour 36–48: the decision table I now keep

Not every batch deserves this ceremony, and the table below is how I decide in under a minute whether the harness pays for itself.

Three checks before I trust any forecast: does the pilot run on the same machine class, does the sample include a cold start, and is the p90 computed from at least sixty successful items? If any answer is no, the forecast is a rumour with decimal places.

What I would repeat, and what I would not

I would repeat the deliberate kill. It cost two minutes and replaced a paragraph of optimism with a number, and it also proved the resume path actually resumes instead of silently restarting from zero. I would repeat pricing at p90 even though it made my estimate look worse, because the estimate exists to protect the morning, not the ego.

I would not repeat two things. First, sampling only when the box is idle, since a pilot measured under no load is a best case wearing a lab coat. Second, checkpointing every tenth item on a job where the expensive part is the model call rather than the disk write — flushing a small JSONL line per item is cheap, and the replay window shrinks by an order of magnitude.

Limitations, and who should skip this

A sixty-item pilot is a smoke test, not a statistics project, and it will mislead you if latency drifts by time of day or if your item mix is not uniform. Free infrastructure is shared by definition, so treat any p90 as a snapshot with a date on it rather than a guarantee. If your job has hard external deadlines, contractual latency, or per-item effects that must occur exactly once, this harness is not enough and you need durable orchestration instead.

Skip this entirely for short jobs, for one-off scripts you will never run again, and for anything where the cost of the pilot exceeds the cost of just running the batch and watching it.

If you want to try the harness cheaply, run the sixty-item pilot and the two forced restarts against a free model and a free server before you commit to the full run — the numbers you get back will tell you more about your own code than any estimate I could hand you.