Triage Nondeterminism Before Golden Tests: A Probe for Messy Repos

A golden test that passes once proves almost nothing. It proves the harness ran. The second run is where messy repos fail: timestamps, temp paths, hash ordering.

Most teams meet this failure as a refactor bug. They revert a harmless change. Then they add sleep and hope.

This post takes the opposite order. Classify nondeterminism first. Pin only what survived classification. Then make the smallest safe change.

The artifact is a stdlib-only probe, a mask budget, and a mask-set growth gate. Both are checkable by CI.

MonkeyCode provides free model access and a free server option (operator-supplied). That matters here for one narrow reason: the probe runs the same command five times. Cheap repeated execution is the whole workload.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The failure mode, stated precisely

A characterization test captures current output. If current output is not a function of input, the capture is not a specification. It is a sample.

Three sources dominate in practice:

1. Clocks. Log lines, cache headers, and created_at fields.
2. Environment. Absolute paths, locale collation, PYTHONHASHSEED set ordering.
3. Allocation. PIDs, UUIDs, ephemeral ports, memory addresses.

Each one is legitimate output. None of it belongs in a golden file.

Step 1: Pick one entrypoint and time it

Characterization tests need a fast, deterministic entrypoint. A CLI command beats a function call here. It exercises the wiring too.

time python -m app.report --id 42 > /tmp/out.txt
wc -l /tmp/out.txt

Rules I apply before writing a single test:

• Under 10 seconds, or you will not run it after every step.
• One process, one exit code, stdout as the artifact.
• No live network. Use a recorded fixture or a local stub.

If the entrypoint cannot meet these, fix that first. Probing a slow command just multiplies the wait.

Step 2: Run the determinism probe

The probe executes the same command under five controlled perturbations. Each run gets a fresh copy of the work tree. Then it classifies output line by line.

The script below is a template, not a battle-tested library. I have not executed it against your repo. Run it on a scratch copy first.

#!/usr/bin/env python3
# determinism_probe.py - stdlib only, Python 3.10+
import argparse, json, os, re, shutil, subprocess, sys, tempfile

PERTURB = [
    ("baseline", {}),
    ("tz", {"TZ": "Pacific/Kiritimati"}),
    ("hashseed-a", {"PYTHONHASHSEED": "0"}),
    ("hashseed-b", {"PYTHONHASHSEED": "4242"}),
    ("locale", {"LC_ALL": "C", "LANG": "C", "TZ": "UTC"}),
]

DEFAULT_PATTERNS = {
    "iso8601": r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})",
    "uuid4": r"[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}",
    "tmppath": r"/tmp/[A-Za-z0-9_./-]{4,}",
    "elapsed": r"\b\d+(?:\.\d+)?(?:ms|us|s)\b",
}

def run(cmd, extra, cwd):
    env = os.environ.copy()
    env.update(extra)
    proc = subprocess.run(cmd, shell=True, cwd=cwd, env=env,
                          capture_output=True, text=True, timeout=180)
    return proc.stdout

def mask(line, patterns):
    used, out = [], line
    for name, pat in patterns.items():
        new = re.sub(pat, f"<{name}>", out)
        if new != out:
            used.append(name)
            out = new
    return out, used

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--cmd", required=True)
    ap.add_argument("--workdir", default=".")
    ap.add_argument("--patterns", default="{}", help="JSON, extra candidates")
    ap.add_argument("--max-masked", type=float, default=0.25)
    ap.add_argument("--out", default="determinism_report.json")
    a = ap.parse_args()

    patterns = dict(DEFAULT_PATTERNS)
    patterns.update(json.loads(a.patterns))

    outputs = []
    for _, extra in PERTURB:
        tmp = tempfile.mkdtemp(prefix="probe-")
        tree = os.path.join(tmp, "repo")
        shutil.copytree(a.workdir, tree, symlinks=True)
        outputs.append(run(a.cmd, extra, tree))
        shutil.rmtree(tmp, ignore_errors=True)

    runs = [o.splitlines() for o in outputs]
    if len({len(r) for r in runs}) != 1:
        print("UNSTABLE: line count drifts", file=sys.stderr)
        sys.exit(2)

    rows, masked = [], 0
    for i in range(len(runs[0])):
        variants = {r[i] for r in runs}
        if len(variants) == 1:
            rows.append({"line": i, "status": "stable"})
            continue
        forms, used = set(), set()
        for v in variants:
            form, names = mask(v, patterns)
            forms.add(form)
            used.update(names)
        if len(forms) == 1 and used:
            masked += 1
            rows.append({"line": i, "status": "volatile",
                         "patterns": sorted(used), "variants": len(variants)})
        else:
            rows.append({"line": i, "status": "unstable",
                         "variants": len(variants),
                         "sample": sorted(variants)[:2]})

    fraction = masked / max(len(rows), 1)
    report = {"cmd": a.cmd, "runs": len(runs), "lines": rows,
              "masked_fraction": round(fraction, 4),
              "max_masked": a.max_masked}
    with open(a.out, "w") as f:
        json.dump(report, f, indent=2)

    unstable = sum(1 for r in rows if r["status"] == "unstable")
    print(f"lines={len(rows)} volatile={masked} unstable={unstable} "
          f"masked_fraction={fraction:.3f}")
    if unstable:
        sys.exit(2)
    if fraction > a.max_masked:
        sys.exit(3)

if __name__ == "__main__":
    main()

Run it, then read the summary line before anything else.

python determinism_probe.py --cmd "python -m app.report --id 42" --workdir .
python -c "import json;r=json.load(open('determinism_report.json'));\
print([x for x in r['lines'] if x['status']!='stable'][:5])"

The exit code is the gate. 0 means every line is stable or masked within budget. 2 means at least one line varies in a way no pattern explains. 3 means you are masking so much that the golden file proves nothing.

Step 3: Let a free model propose patterns, not verdicts

Some repos print shapes the default patterns miss. Composite request IDs. Signed cache keys. Base64 nonces.

This is where free model access is genuinely useful. Feed the model two diffing lines. Ask for candidate regexes only.

Prompt: Here are two outputs of the same command.
Volatile line A: req 7f3c-KJ9x-2 token ttl=59
Volatile line B: req 91ba-LmQ4-8 token ttl=58
Return JSON: {"name": "short_snake_case", "regex": "..."}.
Match the whole varying span. No anchors. No explanation.

Then the probe decides. A candidate is accepted only if applying it collapses every observed variant to one identical line.

python determinism_probe.py --cmd "python -m app.report --id 42" \
  --patterns '{"reqid": "[0-9a-f]{4}-[A-Za-z0-9]{4}-[0-9]"}' \
  --max-masked 0.25

Two hard rules keep this honest:

1. The model never writes assertions, expected values, or normalization functions. It writes candidate patterns only.
2. A generated pattern that does not reduce the variant set is discarded. No pattern is trusted because it looks plausible.

I ran the five perturbed executions on the free server option. Repeated short container runs are exactly the workload where a free tier is enough. The probe is CPU-light and stateless.

Step 4: Turn the report into the golden test

The report already contains the mask list per line. The test imports the same mask function, so production and test cannot drift.

# tests/test_report_golden.py - sketch
import json, subprocess
from pathlib import Path
from determinism_probe import mask, DEFAULT_PATTERNS

REPORT = json.loads(Path("determinism_report.json").read_text())


def normalize(text):
    rows = REPORT["lines"]
    out = []
    for row, line in zip(rows, text.splitlines()):
        if row["status"] == "volatile":
            pats = {k: DEFAULT_PATTERNS[k] for k in row["patterns"]}
            line, _ = mask(line, pats)
        out.append(line)
    return "\n".join(out)


def test_report_is_stable():
    proc = subprocess.run(["python", "-m", "app.report", "--id", "42"],
                          capture_output=True, text=True, check=True)
    assert normalize(proc.stdout) == Path("golden/report.txt").read_text()

Now run the suite three times in a row. It must be green every time. A test that needs luck is not a characterization test.

Step 5: The smallest safe change, with a growth gate

Save determinism_report.json before touching code. Change one thing. Re-run the probe. Compare.

cp determinism_report.json before.json
# apply the smallest safe change
python determinism_probe.py --cmd "python -m app.report --id 42" --workdir .
python - <<'PY'
import json
b = json.load(open('before.json'))['lines']
a = json.load(open('determinism_report.json'))['lines']
before = {r['line']: r.get('patterns', []) for r in b}
after = {r['line']: r.get('patterns', []) for r in a}
grew = [k for k in after if sorted(after[k]) != sorted(before.get(k, []))]
print('mask set changed on lines:', grew)
PY

The gate is simple. After a mechanical refactor, the mask set must not grow. A new volatile line means you moved an allocation, a clock read, or an unordered iteration into observable output. That is a behavior change, not a rename.

When the gate trips, decide explicitly. Either accept the new volatility and re-baseline, or shrink the change until the gate is quiet.

Decision table

Limitations and who should not use this

The probe compares lines by position. If output order is genuinely nondeterministic, it reports unstable and stops. That is intentional. Sort first, probe second.

It also assumes text output on stdout. Do not apply it to binary artifacts, snapshot images, or structured logs split across multiple files. Normalize those separately.

Skip this approach when the masked fraction would exceed a quarter of the output. At that point the golden file is a template, and template diffs catch nothing. Parse the output into fields and assert on the fields instead.

Finally, this does not replace a real test suite. It buys you a trustworthy before-picture for one entrypoint. It says nothing about whether current behavior is correct.

What you have at the end

Three files and two exit codes. A JSON report of what varies. A normalized golden file. A gate that turns a new source of volatility into a failing build.

That is enough to make a messy-repo refactor boring. Boring is the goal.

If you only copy one thing, copy the ordering: classify, pin, then change. If you want to try the loop on a free tier, the probe is small enough to run start to finish on the free server option before you pay for anything.