Hash the Failure Corpus Before You Split a Validator

A validator refactor fails when recorded error bytes drift.
Pin a frozen failure corpus before any structural edit.

Happy-path tests usually miss the real external contract.
Callers observe breakage through exit codes and stderr.
Those two channels cost less than a full suite.

The procedure below is a labeled, unexecuted proposal.
It claims no production metrics and no customer results.
Run every command on a disposable working copy.

Why error bytes beat informal review

Informal review cannot detect silent message punctuation rewrites.
A model edit often changes wrapping, order, and casing.
Downstream scripts then parse the wrong stable text.

Exit status belongs to the public CLI surface.
Stderr bytes also belong to that public surface.
Treat both channels as frozen bytes, not prose.

Do not begin with a god-file class split.
Do not begin with a new exception hierarchy.
Begin with a catalog of known, replayable failures.

What the catalog must freeze

Freeze four fields for every argv failure case.
Store the argument vector as a JSON list.
Store the process exit code as a plain integer.

Hash stdout with SHA-256 after a normalizer.
Hash stderr with SHA-256 after the same normalizer.
Commit both digests beside the argv corpus file.

Strip wall-clock timestamps before hashing either stream.
Strip absolute paths when the tool prints them.
Document each redaction so reviewers can challenge it.

The catalog file becomes the characterization oracle here.
Leave the production module untouched until tests stay green.
Green means every frozen case still matches exactly.

Artifact: replay harness for failure rows

The harness below is unexecuted sample code.
Copy it into tests/test_error_catalog.py before any refactor.
Point TOOL at the current messy CLI entrypoint.

# tests/test_error_catalog.py
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
TOOL = ROOT / "messy_pkg" / "cli.py"
CATALOG = ROOT / "tests" / "fixtures" / "error_catalog.json"
CORPUS = ROOT / "tests" / "fixtures" / "failure_argv.json"


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def normalize(stream: bytes) -> bytes:
    text = stream.decode("utf-8", errors="replace")
    lines = []
    for raw in text.splitlines():
        if raw.startswith("DEBUG ") or raw.startswith("elapsed="):
            continue
        if "/tmp/" in raw or "/var/" in raw or ":\\" in raw:
            raw = "PATH_REDACTED"
        lines.append(raw.rstrip())
    return "\n".join(lines).encode("utf-8")


def run_case(argv: list[str]) -> dict:
    env = {
        **os.environ,
        "TZ": "UTC",
        "LC_ALL": "C.UTF-8",
        "PYTHONHASHSEED": "0",
        "PYTHONUTF8": "1",
    }
    proc = subprocess.run(
        [sys.executable, str(TOOL), *argv],
        cwd=ROOT,
        capture_output=True,
        env=env,
    )
    return {
        "argv": argv,
        "returncode": proc.returncode,
        "stdout_sha256": sha256_bytes(normalize(proc.stdout)),
        "stderr_sha256": sha256_bytes(normalize(proc.stderr)),
    }


def load_json(path: Path):
    return json.loads(path.read_text(encoding="utf-8"))


def test_error_catalog_matches_frozen_bytes():
    corpus = load_json(CORPUS)
    frozen = load_json(CATALOG)
    assert len(corpus) == len(frozen)
    observed = [run_case(item["argv"]) for item in corpus]
    for got, expected in zip(observed, frozen):
        assert got == expected, (got, expected)

Seed the corpus with ugly, real failure argv lists.
Include missing flags, empty files, and invalid encodings.
Include one success row as a control against total silence.

[
  {"argv": ["--strict", ""]},
  {"argv": ["--strict", "not-a-file.json"]},
  {"argv": ["--limit", "-1", "ok.json"]},
  {"argv": ["--limit", "nope", "ok.json"]},
  {"argv": ["ok.json"]},
  {"argv": []}
]

Generate the catalog once from the unedited tree.
Do not regenerate it after you start moving functions.

export TZ=UTC LC_ALL=C.UTF-8 PYTHONHASHSEED=0 PYTHONUTF8=1
python - <<'PY'
import json, sys
from pathlib import Path
sys.path.insert(0, "tests")
from test_error_catalog import CORPUS, CATALOG, load_json, run_case
rows = [run_case(item["argv"]) for item in load_json(CORPUS)]
CATALOG.parent.mkdir(parents=True, exist_ok=True)
CATALOG.write_text(json.dumps(rows, indent=2) + "\n", encoding="utf-8")
print("rows", len(rows), "path", CATALOG)
PY

Then lock the oracle files in git before edits.

git add tests/fixtures/failure_argv.json tests/fixtures/error_catalog.json
git add tests/test_error_catalog.py
git commit -m "test: freeze validator error catalog"
pytest tests/test_error_catalog.py -q

Numbered procedure

1. Copy the messy package onto a throwaway git branch.
2. List every user-visible failure you can still trigger.
3. Encode each failure as an argv list, never as narrative.
4. Normalize volatile lines before hashing captured streams.
5. Write the catalog from the still-unedited working tree.
6. Prove pytest fails after one deliberate stderr string edit.
7. Extract one helper that cannot change any frozen hash.
8. Re-run the catalog and stop when any single row drifts.

Prove the harness actually binds to message text.
A catalog that stays green after a mutation is too weak.

python - <<'PY'
from pathlib import Path
p = Path("messy_pkg/cli.py")
text = p.read_text(encoding="utf-8")
mutated = text.replace("missing file", "file missing", 1)
if mutated == text:
    raise SystemExit("no target string found; pick a real message")
p.write_text(mutated, encoding="utf-8")
PY
pytest tests/test_error_catalog.py -q
git checkout -- messy_pkg/cli.py

If pytest remains green, extend the argv corpus immediately.
Regenerate the catalog only after you understand that gap.
Never regenerate to hide an extract you already wrote.

The smallest safe change

Do not rename the package during this pass.
Do not introduce a new exception type yet.
Extract one pure helper that preserves every digest.

The extract below is labeled pseudocode, not a field result.
Keep stderr wording inside the original CLI module.
Move only input coercion that does not print.

# messy_pkg/flags.py
def normalize_limit(raw: str | None) -> int:
    if raw is None:
        return 100
    try:
        value = int(raw)
    except ValueError as exc:
        raise SystemExit(2) from exc
    if value < 0:
        raise SystemExit(2)
    return value

Call it from the existing branch site only.
Leave help text, log lines, and usage banners untouched.
The catalog must stay byte-identical after the move.

Only the first row is in scope today.
Every other row waits for a later catalog expansion.
Scope control is the whole point of the table.

Where a free coding model fits

A coding model helps only after the catalog stays green.
It cannot replace frozen hashes or the subprocess gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Use that pair on the isolated helper and tests only.
Keep the rest of the god module out of the prompt.

Proposed prompt follows; it is not a recorded session transcript.

Here is normalize_limit plus tests/test_error_catalog.py.
Propose a pure extract that keeps every SHA-256 identical.
Do not rename CLI messages. Do not add exception classes.
Return a unified diff limited to flags.py and one call site.

Treat the returned diff as untrusted text.
Apply it on the same throwaway branch only.
Run pytest tests/test_error_catalog.py before any broader suite.

The free server is for that narrow review loop.
It is not permission to restyle the surrounding package.
Reject diffs that touch help text or log format strings.

Locale, hash seed, and exit mapping

Unset locales make stderr collation change under you.
LC_ALL=C.UTF-8 removes that hidden source of drift.
PYTHONHASHSEED=0 removes randomized set rendering in traces.

SystemExit(n) and os._exit(n) are not interchangeable.
The first still runs finally blocks and flush hooks.
The second can skip atexit handlers and lost stderr tails.

Do not “fix” that mapping during the first extract.
Record the current mapping as part of the catalog contract.
Change process teardown only after a dedicated corpus exists.

Windows and POSIX may disagree on signal-style codes.
This harness records the integer your current platform returns.
Do not compare catalogs across operating systems without a note.

Limitations

This oracle ignores runtime and peak memory entirely.
It ignores log files written outside captured stderr.
It ignores exit codes from grandchild helper processes.

SHA-256 equality rejects harmless wording improvements too.
That rejection is the characterization gate working as designed.
Collect a second corpus if you truly intend to reword.

Normalization can hide real path-leak bugs as well.
Redacting every path may swallow a disclosure you needed.
Review redaction rules with the same care as production code.

Subprocess replay is slower than ordinary unit tests.
Keep the corpus to a few dozen high-value rows.
Split huge tools by subcommand instead of one giant catalog.

Who should skip this approach

Skip this when the tool has no stable CLI contract.
Skip this when failures appear only as GUI dialogs.
Skip this when legal copy must change in the same week.

Do not use the catalog to justify deleting typed tests.
Do not use it as a substitute for schema validation.
Do not point it at unsanitized, untrusted fixture payloads.

Security parsers still need adversarial generation later.
A frozen catalog is not a fuzzer and not a linter.
Add fuzzing only after these hashes are already stable.

Closing checklist

Confirm the catalog JSON is committed on the branch.
Confirm pytest fails on a one-word stderr mutation.
Confirm the extract touches one helper and one call site.
Confirm every digest matches before you merge anything.

Hash the failures before any structural split begins.
Move one pure function only after that gate holds.
If a free model loop is used, attach the catalog to the same review.