DEV Community

Dakota Huang
Dakota Huang

Posted on

Count the Clocks Before Characterization Tests: A Nondeterminism Census for Messy Repos

#testing #refactoring #python #ai

Characterization tests fail for a boring reason: the code reads the wall clock.
You cannot pin behavior you cannot reproduce twice.
So count the clocks first, then move code.

There is a second trap. unittest.mock.patch targets a dotted path, not an object. A refactor that moves a module silently breaks the patch. The test still passes. The freeze silently stops working.

This article covers one artifact that catches both problems: a nondeterminism census with a patch-target inventory.

Step 1: Census every clock, seed, and host read

Run this on the repo root. It parses ASTs, so it does not need imports to succeed.

#!/usr/bin/env python3
# nondet_census.py - list every clock, seed, and host read in a repo.
import ast
import json
import pathlib
import sys

WATCH = {
    "time": {"time", "monotonic", "perf_counter", "time_ns"},
    "datetime": {"now", "utcnow", "today"},
    "random": {"random", "randint", "choice", "shuffle"},
    "uuid": {"uuid1", "uuid4"},
    "os": {"urandom", "getpid", "getenv"},
    "secrets": {"token_hex", "token_bytes", "randbelow"},
    "socket": {"gethostname", "gethostbyname"},
}
SKIP = {".git", ".venv", "venv", "build", "dist", "node_modules"}


def module_name(path, root):
    parts = list(path.relative_to(root).with_suffix("").parts)
    if parts[-1] == "__init__":
        parts.pop()
    return ".".join(parts)


def scan(path, root):
    tree = ast.parse(path.read_text(encoding="utf-8"))
    mod = module_name(path, root)
    aliases, rows = {}, []

    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for a in node.names:
                if a.name in WATCH:
                    aliases[a.asname or a.name] = a.name

    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom) and node.module in WATCH:
            for a in node.names:
                if a.name in WATCH[node.module]:
                    rows.append({
                        "file": str(path), "line": node.lineno,
                        "kind": "local_copy",
                        "symbol": f"{node.module}.{a.name}",
                        "patch_target": f"{mod}.{a.asname or a.name}",
                    })
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
            owner = node.func.value
            if isinstance(owner, ast.Name) and owner.id in aliases:
                real = aliases[owner.id]
                if node.func.attr in WATCH[real]:
                    rows.append({
                        "file": str(path), "line": node.lineno,
                        "kind": "module_attr",
                        "symbol": f"{real}.{node.func.attr}",
                        "patch_target": f"{mod}.{owner.id}.{node.func.attr}",
                    })
    return rows


if __name__ == "__main__":
    root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
    out = []
    for path in sorted(root.rglob("*.py")):
        if SKIP & set(path.parts):
            continue
        out.extend(scan(path, root))
    pathlib.Path("nondet_targets.json").write_text(json.dumps(out, indent=2))
    print(f"{len(out)} rows -> nondet_targets.json", file=sys.stderr)
Enter fullscreen mode Exit fullscreen mode 
python nondet_census.py .
# 61 rows -> nondet_targets.json
Enter fullscreen mode Exit fullscreen mode

Sixty-one rows is normal. Most are harmless. You need the list anyway, because the freeze step consumes it.

Step 2: Sort rows by what the patch can actually reach

Each row is one of three shapes. The shape decides your first move.

Row kind Example Patch target First move
module_attr import time; time.time() app.reports.build.time.time Patch at the importing module
local_copy from time import time app.reports.build.time Patch the local name
server_side SELECT now(), C extension clocks none Pin inputs and outputs instead

The local_copy row is the quiet one. Patching time.time does nothing if the module did a from-import. Your test looks deterministic and is not.

Rows also cluster. Ten call sites in one module usually mean one seam, not ten fixes. Group by file before you touch anything.

The classification is mechanical, and a language model is decent at it. I paste the JSON rows into MonkeyCode's free model access with a strict output schema: {line, bucket, reason}, bucket in freeze|inject|leave.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the assertions. The model only labels rows. Every label gets a human yes before it enters the target file.

Step 3: Freeze from the inventory, not from memory

Your conftest.py should read the census output. Hand-written patch lists drift in a week.

# conftest.py
import json
import os
import pathlib
import random

import pytest

ROWS = json.loads(pathlib.Path("nondet_targets.json").read_text())
SEED = int(os.environ.get("GATE_SEED", "0"))
FROZEN = 1_700_000_000.0


@pytest.fixture(autouse=True)
def frozen_world(monkeypatch):
    random.seed(SEED)
    for row in ROWS:
        if row["kind"] == "module_attr":
            monkeypatch.setattr(row["patch_target"], lambda: FROZEN)
        elif row["kind"] == "local_copy":
            monkeypatch.setattr(row["patch_target"], lambda: FROZEN)
    yield
Enter fullscreen mode Exit fullscreen mode

monkeypatch.setattr accepts dotted strings and imports the module for you. See the pytest docs on monkeypatch.

One caveat: datetime.now() needs a datetime return value. Branch on row["symbol"] and return datetime.fromtimestamp(FROZEN) instead of a float.

Step 4: Add a test that the patch still lands

This is the gate that survives a refactor. It resolves every target and checks object identity against the real stdlib function.

# tests/test_patch_targets.py
import importlib
import json
import pathlib
import time as stdlib_time

import pytest

ROWS = json.loads(pathlib.Path("nondet_targets.json").read_text())
STDLIB = {
    "time.time": stdlib_time.time,
    "time.monotonic": stdlib_time.monotonic,
    "time.perf_counter": stdlib_time.perf_counter,
}


def resolve(dotted):
    mod, _, attr = dotted.rpartition(".")
    obj = importlib.import_module(mod)
    for part in attr.split("."):
        obj = getattr(obj, part)
    return obj


@pytest.mark.parametrize("row", ROWS, ids=lambda r: r["patch_target"])
def test_target_resolves(row):
    try:
        assert resolve(row["patch_target"]) is not None
    except (ImportError, AttributeError) as exc:
        pytest.fail(f"patch target gone: {row['patch_target']} ({exc})")


@pytest.mark.parametrize("row", [r for r in ROWS if r["kind"] == "module_attr"])
def test_patch_hits_stdlib_object(row):
    assert resolve(row["patch_target"]) is STDLIB[row["symbol"]]
Enter fullscreen mode Exit fullscreen mode

Now a module move produces a red test with the old dotted path in the name. That is the failure you want. Compare with the unittest.mock docs on where patches are applied: docs.python.org.

Step 5: Run the stability gate

Twenty seeds, fixed hash seed, same command. Flaky characterization tests are worse than none.

#!/usr/bin/env bash
set -euo pipefail
mkdir -p runs
for seed in $(seq 1 20); do
  GATE_SEED=$seed PYTHONHASHSEED=$seed \
    pytest -q tests/characterization --tb=no > "runs/seed_$seed.txt" \
    || { echo "nondeterministic at seed=$seed"; exit 1; }
done
echo "20/20 stable"
Enter fullscreen mode Exit fullscreen mode

The loop is CPU-light and wall-clock heavy. Twenty runs of a slow suite is hours on a laptop. This is where a hosted runner earns its keep, and MonkeyCode's free server option is one place to put it. Treat that as an option, not a benchmark.

Save the runs/ folder. It is the baseline you diff against after the change.

Step 6: Make the smallest safe change, then re-run three things

Do the extraction. Then run, in this order:

  1. python nondet_census.py . and diff the new JSON against the old one.
  2. pytest -q tests/test_patch_targets.py.
  3. The 20-seed loop again.

A clean diff means no clock moved into a new module. A failing target test means a patch path died. A flaky seed means you moved code across a freeze boundary.

If the census diff shows a target count change, stop. That is a behavior change hiding inside a move.

Limitations and who should skip this

  • Child processes do not inherit patches. Subprocesses that call the clock need env-var injection.
  • C extensions and database now() are invisible to AST scans. Pin their inputs and outputs instead.
  • Threads plus global monkeypatching can race. Prefer an injected clock object in threaded code.
  • PYTHONHASHSEED fixes hash order only. It says nothing about set iteration in other runtimes.
  • If the repo has no runnable test suite, the census still helps, but characterization comes first.

Skip this if your nondeterminism is the specification. Fuzzers and property tests want randomness, not a frozen clock.

Also skip it if nobody will review the model's row labels. An unreviewed leave bucket is an unfrozen test waiting to flake in CI.

What this buys you

Characterization tests are only as stable as the clocks under them. A census makes that stability checkable, and a patch-target test makes it survive the refactor.

The script is the part worth stealing. Run it on your messiest module before you promise anyone a safe change.

If you want a second opinion on row classification, the free model access is enough for that job. Keep the tests yours.

Top comments (0)