Capture the Process Transcript Before You Extract One Helper

A messy refactor fails at the process boundary first. Characterization tests should lock that boundary before any extract. Change one private helper only after the transcript matches.

Internal cleanliness does not prove external safety for callers. Callers care about exit codes, stdout, and stderr. Those three streams are the real public surface of many CLIs.

Why the first extract breaks CI

AI diffs often look locally correct and still break scripts. A renamed helper can still change published help text. A reordered branch can flip an exit code.

Downstream CI then fails on a supposedly safe extract. This pattern is common in mixed-age Python packages. One module owns parsing, I/O, and formatting together.

Existing tests, if any, assert internals nobody else uses. Those tests stay green while the CLI contract drifts. The missing oracle is the launched process itself.

What a process transcript records

A process transcript is a frozen subprocess contract. It stores argv, selected env keys, and cwd. It also stores exit code, stdout, and stderr.

Optional file digests cover side effects the CLI writes. The transcript is not a private unit test. It is not a formatter or style check.

It is a characterization oracle for one entrypoint. The snippets below are a proposed local harness. They are unexecuted examples for a messy tree.

Artifact: a transcript harness

The harness lives beside the messy package root. It does not import private helpers at all. It only launches the published command from subprocess.

1. Pin the command matrix

Create transcripts/cases.json next to the package. Keep the matrix small, public, and secret-free.

{
  "entrypoint": ["python", "-m", "messypkg"],
  "cases": [
    {"id": "help", "argv": ["--help"], "env": {}},
    {"id": "missing-input", "argv": ["run"], "env": {}},
    {
      "id": "sample-ok",
      "argv": ["run", "--in", "fixtures/ok.json"],
      "env": {"MESSY_MODE": "strict"}
    },
    {
      "id": "sample-bad",
      "argv": ["run", "--in", "fixtures/bad.json"],
      "env": {"MESSY_MODE": "strict"}
    }
  ]
}

Four cases already catch help text and usage errors. They also catch one success path and one validation failure. Add a case only when a real caller depends on it.

2. Pin locale, timezone, and hash seed

Locale can rewrite help text and error messages. Hash randomization can shuffle debug dumps of sets. Timezone can shift any datetime printed to stdout.

Pin these values inside the capture runner process. Treat them as part of the contract, not decoration.

# proposed: pins inside transcripts/capture.py
env["PYTHONHASHSEED"] = "0"
env["TZ"] = "UTC"
env["LC_ALL"] = "C"
env["LANGUAGE"] = "C"

Without those pins, goldens become machine-specific and flaky. Two laptops then disagree on a “pure” extract. The disagreement is the environment, not the helper.

3. Normalize volatile output

Timestamps, absolute paths, and durations must not enter goldens. Strip them before hashing or writing files. Normalization is part of the oracle contract.

# proposed: transcripts/normalize.py
import re

PATH_RE = re.compile(r"(?:/|[A-Z]:\\)[^\s]+")
TIME_RE = re.compile(r"\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\b")
DUR_RE = re.compile(r"\b\d+\.\d{2,}s\b")

def normalize(text: str, cwd: str) -> str:
    text = text.replace(cwd, "<CWD>")
    text = PATH_RE.sub("<PATH>", text)
    text = TIME_RE.sub("<TS>", text)
    text = DUR_RE.sub("<DUR>", text)
    return text.replace("\r\n", "\n")

Skip this step and every machine will flake. Over-strip paths and a broken relative load can hide. Review each regex before trusting the goldens.

4. Capture one transcript per case

# proposed: transcripts/capture.py
import hashlib, json, os, subprocess, sys
from pathlib import Path
from normalize import normalize

ROOT = Path(__file__).resolve().parents[1]
GOLD = ROOT / "transcripts" / "goldens"

def run_case(case, entrypoint):
    env = os.environ.copy()
    env.update(case.get("env") or {})
    env["PYTHONHASHSEED"] = "0"
    env["TZ"] = "UTC"
    env["LC_ALL"] = "C"
    env["LANGUAGE"] = "C"
    proc = subprocess.run(
        entrypoint + case["argv"],
        cwd=ROOT,
        env=env,
        capture_output=True,
        text=True,
        timeout=30,
    )
    stdout = normalize(proc.stdout, str(ROOT))
    stderr = normalize(proc.stderr, str(ROOT))
    return {
        "id": case["id"],
        "argv": case["argv"],
        "exit": proc.returncode,
        "stdout": stdout,
        "stderr": stderr,
        "stdout_sha": hashlib.sha256(stdout.encode()).hexdigest()[:16],
        "stderr_sha": hashlib.sha256(stderr.encode()).hexdigest()[:16],
    }

def main(write=False):
    spec = json.loads((ROOT / "transcripts" / "cases.json").read_text())
    GOLD.mkdir(parents=True, exist_ok=True)
    failures = []
    for case in spec["cases"]:
        got = run_case(case, spec["entrypoint"])
        path = GOLD / f"{case['id']}.json"
        if write or not path.exists():
            path.write_text(json.dumps(got, indent=2) + "\n")
            continue
        want = json.loads(path.read_text())
        if got != want:
            failures.append(case["id"])
            print(f"MISMATCH {case['id']}")
            print("exit", want["exit"], "->", got["exit"])
            print("stdout_sha", want["stdout_sha"], "->", got["stdout_sha"])
            print("stderr_sha", want["stderr_sha"], "->", got["stderr_sha"])
    if failures:
        sys.exit(1)

if __name__ == "__main__":
    main(write="--write" in sys.argv)

The comparison is exact JSON equality after normalize. Any drift in exit, stdout, or stderr fails. That failure is the extract’s stop condition.

5. Record goldens, then refuse application edits

python transcripts/capture.py --write
git add transcripts/cases.json transcripts/goldens transcripts/*.py
git commit -m "test: pin process transcripts before extract"

Do not edit application code in the same commit. The oracle must land first on main. Later diffs then have a binary baseline.

6. Re-run on every extract attempt

python transcripts/capture.py

A green run means the process contract held. A red run means the extract is not small enough. Revert before stacking a second structural change.

Read a mismatch without guessing

Dump the new transcript beside the golden file. Diff those two JSON documents, not the helper. The three fields tell you which seam moved.

python - <<'PY'
import json, subprocess, sys
from pathlib import Path
sys.path.insert(0, "transcripts")
from capture import run_case
spec = json.loads(Path("transcripts/cases.json").read_text())
case = next(c for c in spec["cases"] if c["id"] == "sample-ok")
got = run_case(case, spec["entrypoint"])
Path("/tmp/got-sample-ok.json").write_text(json.dumps(got, indent=2) + "\n")
PY
diff -u transcripts/goldens/sample-ok.json /tmp/got-sample-ok.json

Exit-only drift means control flow changed. Stdout-only drift means a formatter changed. Stderr-only drift means logging or validation changed.

Both hashes changing means the extract was too large. Stop and revert that patch immediately. Do not negotiate with a two-stream failure.

Decision table: what to change next

One row, one action, one commit. Stacked changes hide the failing seam. The table is the review script for the patch.

The smallest safe change

After goldens exist, extract one helper only. Keep the public command identical in argv. Move a pure function, not I/O, first.

Example shape, labeled as a proposal only:

# before: messypkg/cli.py owns everything
def run(argv):
    raw = load(argv.input)
    if "items" not in raw:
        print("missing items", file=sys.stderr)
        return 2
    print(json.dumps({"count": len(raw["items"])}))
    return 0

# after: one helper, same control flow
def count_items(raw):
    if "items" not in raw:
        return None
    return len(raw["items"])

def run(argv):
    raw = load(argv.input)
    n = count_items(raw)
    if n is None:
        print("missing items", file=sys.stderr)
        return 2
    print(json.dumps({"count": n}))
    return 0

The extract changes no bytes on the three streams. Transcripts stay equal under capture.py. That equality is the definition of small.

If count_items later needs a message change, split commits. Update goldens in a dedicated behavior commit. Do not mix behavior change with structure change.

Fixture rules that keep the oracle honest

Fixtures must be tiny, committed, and secret-free. The case matrix must not touch the network. Timeouts protect the oracle from accidental hangs.

{"items": [{"id": 1}, {"id": 2}]}

{"title": "not-a-list"}

Two files cover the success path and the missing-key path. That is enough to gate one helper extract. More fixtures belong to later product work, not this gate.

Keep timeout=30 until a case proves it is too low. Long jobs need a different oracle, such as sampled logs. Do not raise the timeout to hide hangs.

Where a coding model fits

A model is useful after the oracle exists. It is not useful as the first reader of a god file. Prompt it with goldens, the helper name, and the mismatch table.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two facts matter only as a place to run the proposal step. They do not replace transcripts/capture.py.

They also do not prove the extract is safe. Keep the model off the capture script itself. Let it suggest the helper body, then re-run the harness locally.

Reject any patch that fails a golden file. A free server is optional compute for that loop. The contract still lives in git if the server is down.

If you already pin transcripts, a free model pass is optional.

Limitations

This oracle ignores private call graphs entirely. It will miss a leaky helper unused by the CLI. It will miss performance regressions under thirty seconds.

It will miss Unicode edge cases absent from fixtures. Normalization can hide real path bugs. Over-stripping may allow a broken relative load.

JSON equality is brittle with unordered maps on some runtimes. Python 3.7+ preserves insertion order in dict dumps. Other languages may need canonical encoding.

The harness uses a 30-second timeout on purpose. That bound is a safety rail, not a benchmark. Do not treat a green transcript as a load test.

Who should not use this

Do not use this on libraries with no process boundary. Public functions need API-level characterization, not subprocess goldens. A subprocess.run oracle will not see those callers.

Do not use this as a substitute for typed contracts. Greenfield services should write real tests first. Characterization is for inherited mess, not new design.

Do not use this to justify a large rewrite. The method forbids stacked extracts by design. If the module cannot yield one helper, stop.

Do not send secrets into golden files. Redact tokens inside normalize(). Keep every fixture synthetic and committed.

Checklist before merge

1. Goldens committed on main before the extract branch.
2. One helper moved, with no flag, message, or exit change.
3. python transcripts/capture.py exits 0 on the branch.
4. git diff --stat shows the helper plus transcript tests only.
5. Behavior commits, if any, stay separate from structure commits.

The process transcript is the refactor gate. Structure work that preserves it is allowed. Structure work that perturbs it is a product change.

Treat those as different jobs with different reviews. The extract is finished when the three streams match. Anything else is a new feature, not a refactor.