Lock argv, stdin, and exit codes before one extract

A messy CLI refactor fails at the process boundary.
Internal helper names may change without hurting callers.
Pinned exit codes and parsed stderr cannot drift.

Pin argv, env, stdin, stdout, stderr, and cwd first.
Then extract one function and stop after that change.

The failure mode

Most AI diffs look clean in the editor.
The binary still prints a different error line.
Callers that parse stderr then break in CI.

Characterization at the process edge catches that drift.
Unit tests on private internals often miss it.
The operating system is the real public API.

A library test can pass after a silent flag rename.
A wrapper script then fails on the next deploy.
Process I/O is the contract other teams actually consume.

What you pin

You pin six process values for every scenario.
They are argv, env, stdin, cwd, stdout, and stderr.
You also pin the numeric process exit code.

Do not pin private function names in gold files.
Do not pin log timestamps unless callers parse them.
Do not pin import order inside the package.

Treat documented file writes as contract too.
Treat cache files under /tmp as non-contract noise.
When unsure, leave the path out of gold.

Proposed artifact

The harness below is a proposal, not a measured run.
Copy the script into tools/cli_oracle.py as a start.
Keep the gold files under oracles/cli/ after recording.

It runs the current tree as a subprocess.
It writes a canonical JSON record per scenario.
It diffs that record against a frozen gold file.

Python's subprocess module is the runner.
Do not shell-out through os.system here.
A list argv avoids quoting bugs in gold.

#!/usr/bin/env python3
"""Process-boundary oracle for a messy CLI.

Proposed harness. Treat results as unexecuted until you run it.
"""
from __future__ import annotations

import json
import os
import re
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
GOLD = ROOT / "oracles" / "cli"
SCENARIOS = ROOT / "oracles" / "scenarios.json"

VOLATILE = (
    re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}"),
    re.compile(r"pid=\d+"),
)


def stabilize(text: str) -> str:
    for pat in VOLATILE:
        text = pat.sub("<stable>", text)
    return text


def run_one(spec: dict) -> dict:
    env = os.environ.copy()
    env.update(spec.get("env") or {})
    env.setdefault("TZ", "UTC")
    env.setdefault("LANG", "C")
    env.setdefault("PYTHONHASHSEED", "0")
    proc = subprocess.run(
        spec["argv"],
        input=spec.get("stdin", ""),
        cwd=str(ROOT / spec.get("cwd", ".")),
        env=env,
        text=True,
        capture_output=True,
        timeout=spec.get("timeout_sec", 15),
    )
    return {
        "name": spec["name"],
        "argv": spec["argv"],
        "exit_code": proc.returncode,
        "stdout": stabilize(proc.stdout),
        "stderr": stabilize(proc.stderr),
    }


def canonical(record: dict) -> str:
    return json.dumps(record, indent=2, sort_keys=True) + "\n"


def main(mode: str) -> int:
    specs = json.loads(SCENARIOS.read_text())
    GOLD.mkdir(parents=True, exist_ok=True)
    failed = 0
    for spec in specs:
        record = run_one(spec)
        path = GOLD / f"{spec['name']}.json"
        text = canonical(record)
        if mode == "record":
            path.write_text(text)
            print(f"recorded {path}")
            continue
        if not path.exists():
            print(f"missing gold: {path}", file=sys.stderr)
            failed += 1
            continue
        if path.read_text() != text:
            print(f"drift: {spec['name']}", file=sys.stderr)
            failed += 1
    return 1 if failed else 0


if __name__ == "__main__":
    mode = sys.argv[1] if len(sys.argv) > 1 else "check"
    raise SystemExit(main(mode))

Pair it with a small scenario file.
Keep scenario names stable for gold filenames.
Start with help, error, and stdin paths.

[
  {
    "name": "help_exits_zero",
    "argv": ["python", "-m", "messy_tool", "--help"],
    "stdin": ""
  },
  {
    "name": "missing_file_exits_two",
    "argv": ["python", "-m", "messy_tool", "parse", "no-such.json"],
    "stdin": ""
  },
  {
    "name": "stdin_json_roundtrip",
    "argv": ["python", "-m", "messy_tool", "parse", "-"],
    "stdin": "{\"id\": 1, \"name\": \"alpha\"}\n"
  }
]

This article treats the CLI as text.
Binary stdin needs a separate recorder path.
Do not encode unknown bytes as UTF-8 guesses.

Numbered workflow

Follow these eight steps in strict order.
Do not skip recording the gold files first.

1. Inventory entrypoints

List every console script and __main__ module.
Ignore private helpers until the gold files exist.
Write the list in oracles/entrypoints.md for review.

2. Write few scenarios

Write three to seven scenarios, not a full suite.
Cover help, one happy path, and one error path.
Add a stdin case if the tool reads -.

3. Freeze environment keys

Freeze the environment keys that change output.
Set TZ, LANG, and PYTHONHASHSEED in the harness.
Document any extra keys your tool reads.

4. Record gold files

Record gold files on an unchanged tree.
Run python tools/cli_oracle.py record from the repo root.
Commit the oracles/ directory as its own commit.

5. Prove the check is green

Run the check on the same commit.
Run python tools/cli_oracle.py check before any edit.
The check must pass before any extract.

6. Choose one extract

Choose one extract that cannot change I/O.
Rename a parser helper or split a branch.
Do not change flag names or error text.

7. Re-run the check

Re-run the check after that single extract.
If gold files drift, revert the extract.
Do not rewrite gold to match a new guess.

8. Only then use a remote model

Only then consider a remote coding model.
Feed the oracle and the one target file.
Reject any patch that fails the check command.

Decision table

Use this table before you accept a patch.
Contract columns belong in the gold files.
Internal columns must stay out of the oracle.

If the table says yes, add a gold file.
If the table says no, leave it out.
Mixed signals mean you need a narrower scenario.

Stabilize volatile bytes

Some CLIs stamp time and pid on stderr.
Those bytes are not the product contract.
Strip them before you write gold files.

Apply stabilize() to stdout and stderr in run_one.
Do not strip error codes or flag names.
If you strip too much, the oracle goes blind.

Add a new pattern only with a written reason.
Put that reason beside the regex in the script.
Unexplained filters hide real contract breaks.

Git sequence

Put the oracle in its own commit.
Do not mix recording with the extract.
Later bisect runs depend on that split.

git status --short
git add tools/cli_oracle.py oracles
git commit -m "test: pin CLI process oracle"
# edit one helper only
python tools/cli_oracle.py check
git add -u
git commit -m "refactor: extract parse_object from parse"

Stop if check prints drift.
Do not amend the oracle commit away.
History is part of the characterization record.

When the check fails

When check fails, read the two JSON files.
Diff stdout, stderr, and exit_code before argv.
Argv drift means the scenario file changed.

Exit code drift is a breaking change.
Stdout drift breaks parsers in other repos.
Stderr drift breaks operators who grep logs.

A proposed triage order is below.
It is a checklist, not a measured study.
Walk it before you reopen the editor.

1. Confirm you did not rerun record by accident.
2. Confirm TZ and LANG still match the harness.
3. Confirm the extract touched one module only.
4. Revert if more than one of those checks fail.

Where a remote free model fits

A local oracle is still the source of truth.
A remote model can propose the one extract.
It cannot decide which bytes are the contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option.
Use that remote loop only after gold files exist.
Send the oracle script, the scenario file, and one module.

Keep secrets out of the tree you send.
Do not upload production env files.
The process oracle should run on your machine.

Proposed remote prompt, unexecuted:

The process oracle is tools/cli_oracle.py.
Gold files live in oracles/cli/.
Change only lib/messy_tool/parser.py.
Do not change argv, exit codes, stdout, or stderr.
Stop after one function extract.
I will run: python tools/cli_oracle.py check

If the check fails, discard the patch.
Do not widen the diff to silence the oracle.
The model is a draft tool, not a witness.

Limitations

This harness ignores timing and memory.
It will not catch deadlocks well.
A 15 second timeout is a coarse net.

JSON canonicalization hides key order in records.
It does not hide key order in stdout.
If your CLI prints unordered dicts, freeze that path first.

Subprocess tests will not isolate C extensions.
They will not mock the network without fixtures.
Do not treat three scenarios as full coverage.

Gold files rot when you intend a user-visible change.
Then you must record again on purpose.
Accidental record runs destroy the oracle.

This article does not claim runtime numbers.
It does not rank models or hosts.
The check command is the only pass signal.

Who should not use this

Skip this if your product is only a library API.
Use import-level characterization instead.
Process I/O will not see your public functions.

Skip this if every flag change is already versioned.
A changelog and semver may already be the contract.
The harness then duplicates release work.

Skip this if the CLI talks to live third-party systems.
You need recorded HTTP fixtures first.
Otherwise gold files capture the network, not your code.

Skip this if you cannot run the tool locally.
A remote-only workflow without check is unsafe.
A free server does not replace the oracle.

What smallest safe change means

Small means one extract or one rename.
Safe means the process record is byte-identical.
If both are true, merge the commit.

If the extract needs a flag change, stop.
That is a product change, not a refactor.
Split it into a later, documented commit.

If two helpers look tangled, extract neither yet.
Add one more scenario that splits their behavior.
Then extract the helper the new gold isolates.

Close

Lock the process boundary before you touch names.
Three gold files beat a confident diff.
Run the check after every extract.

Keep the oracle in your repo.
Use a remote model only as a bounded draft.
If the check is red, the extract is wrong.

If you try the harness, keep the check local even when the extract is drafted elsewhere.