The Fixture Moved With the Code: Pinning Test Inputs Before an Agent Patch Rewrites Its Own Oracle

A green suite is only as trustworthy as the inputs it was given. If an agent patch can edit the fixtures, golden files, snapshots, or property-test seeds, then "tests pass" describes the patch's own editing decisions as much as it describes the fix. The gate I describe here hashes test inputs at two git refs, classifies every change, and refuses to accept a flaky-test freeze that does not carry the fixture digest and the seed it was frozen against.

This is not a theory of correctness. It is a cheap, mechanical check that catches a specific class of dishonest green: the patch that changed the oracle instead of the code.

Why freeze records without a digest are unfalsifiable

Most flaky-test freezes I have seen record a test ID, a date, and a reason string. Six weeks later nobody can tell whether the frozen test still checks the same thing. A freeze is a claim about a test's behavior, and behavior depends on inputs. Store the input hash and the seed alongside the reason, or the freeze is an assertion with no witness.

1. Hash test inputs at a git ref, not in your working tree

Read blobs directly out of git so the check does not depend on which branch is checked out. This script uses only the standard library.

#!/usr/bin/env python3
# fixture_manifest.py <base-ref> <patch-ref>
import hashlib, json, subprocess, sys

PATTERNS = ("tests/fixtures/", "tests/golden/", "tests/data/", "__snapshots__/")
SUFFIXES = (".snap", ".golden")

def git(*args):
    return subprocess.run(["git", *args], check=True,
                          capture_output=True).stdout

def inputs_at(ref):
    names = git("ls-tree", "-r", "--name-only", ref).decode().splitlines()
    return [p for p in names
            if p.startswith(PATTERNS) or p.endswith(SUFFIXES)]

def sha(ref, path):
    return hashlib.sha256(git("cat-file", "blob", f"{ref}:{path}")).hexdigest()

def manifest(ref):
    return {p: sha(ref, p) for p in inputs_at(ref)}

base, patch = sys.argv[1], sys.argv[2]
a, b = manifest(base), manifest(patch)
changes = []
for p in sorted(set(a) | set(b)):
    kind = ("removed" if p not in b else
            "added" if p not in a else
            "modified" if a[p] != b[p] else None)
    if kind:
        changes.append({"path": p, "kind": kind,
                        "base": a.get(p), "patch": b.get(p)})
print(json.dumps({"base": base, "patch": patch, "changes": changes}, indent=2))

Run it against the merge base and the patch head: python fixture_manifest.py $(git merge-base main HEAD) HEAD. An empty changes array means the patch did not touch its own inputs, and the rest of this workflow can be skipped.

2. Classify each change against the production diff

A changed fixture is not automatically bad. A new fixture that supports a new test is normal engineering. The signal is the combination of factors, so classify on three axes and only escalate the ambiguous cells.

The two block rows are the reason this gate earns its keep. A patch that rewrites a golden file while touching zero lines of production code is a patch that redefined passing and called it a fix.

3. Check whether the changed input is still referenced

A removed fixture with live references is a latent failure that your current run may not reach. Resolve consumers before you accept the deletion.

python fixture_manifest.py origin/main HEAD \
  | jq -r '.changes[] | select(.kind=="removed") | .path' \
  | while read -r f; do
      stem=$(basename "$f")
      hits=$(grep -rlF "$stem" tests/ --include='*.py' --include='*.ts' || true)
      [ -z "$hits" ] && echo "OK    $f" || echo "LIVE  $f -> $hits"
    done

Treat the LIVE lines as review items, not hard failures. Grep over test sources is a coarse instrument; it will miss dynamic path construction and produce false hits on generic names like input.json. Coarse is fine here, because the alternative is nothing.

4. Pin property-test seeds inside the patch tree

Property tests have the same problem in a different shape. An agent can make a failing property pass by changing the seed, the example budget, or the derandomization profile. None of those changes look like a behavior change in a diff review.

# tests/test_encoding.py
from hypothesis import given, settings, strategies as st

SEED = 20261026  # pinned: reviewed in PR, not derived at runtime

@settings(max_examples=200, derandomize=True, deadline=None)
@given(st.binary(min_size=1, max_size=64))
def test_roundtrip_is_identity(blob):
    assert decode(encode(blob)) == blob

If your property framework accepts an explicit seed, pass the constant. If it only exposes a determinism switch, like Hypothesis's derandomize, use that and record which mode you are in, because the two are not interchangeable. Pinning has a real cost: you stop discovering new counterexamples on every CI run. Schedule an unpinned nightly job if that discovery matters to you.

5. Put the digest and the seed in the freeze record

A freeze record becomes checkable once it names what it froze.

{
  "test_id": "tests/test_ingest.py::test_retry_window",
  "reason": "timing-dependent on shared CI runner; see #4121",
  "observed_flake_rate": "3/40 local runs",
  "fixture_digest": "sha256:9f2c…",
  "seed": 20261026,
  "expires": "2026-11-25",
  "owner": "@datacpp_8185"
}

The gate logic is short: a freeze whose fixture_digest does not match the manifest hash of the inputs that test consumes is stale, and stale freezes fail closed. That single rule prevents the most common decay mode, where a test is frozen, its fixtures are later rewritten, and the freeze silently outlives the thing it was protecting.

Where a free model endpoint and a free server actually helped

The gate above is deterministic, so a model is not needed for the accept/block rows. It is useful for exactly one bucket: the modified fixture changes that touch production code. Those need a human-readable summary of what moved, and reading golden-file diffs is tedious enough that reviewers skip it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My workflow sends only the manifest rows and the diff hunk headers — never the repository — to MonkeyCode's free model access, and asks for a one-paragraph description of what expectation changed. The output is attached to the PR as a comment and is explicitly labeled as machine-generated, unverified. Separately, the free server option let me run fixture_manifest.py on the same host as the agent that produces patches, so the gate runs before the branch is pushed rather than in a queue behind it. Both are operator-supplied availability claims; check the current limits on the product's own pages before you build anything that depends on them being permanent.

Nothing in steps 1 through 5 requires a model. Delete the product and the gate still works.

Limitations and failure modes

• Semantic equivalence is invisible to a hash. Reformatting a golden file trips the gate; a hand-edited value that happens to keep tests passing does not.
• Inputs constructed at runtime from env vars, network calls, or clocks are not in the manifest. This check covers files, not ambient state.
• Snapshot regeneration workflows produce large legitimate diffs. Route them through the review row, not the block row, or the gate will be disabled within a week.
• A green result from this gate means the patch did not move its own inputs. It does not mean the assertions are correct, and it does not mean the properties are meaningful.

Who should not use this

Skip it if your suite has no fixture, golden, or snapshot directories — there is nothing to hash and the classification table collapses to one row. Skip it if snapshot churn is a normal part of your development loop; the review load will exceed the value. Skip it if you cannot pin seeds, because a freeze record without a seed is back to being unfalsifiable.

If you already have a merge gate, the smallest useful version is one script and one rule: no patch may block-classify a fixture change, and no freeze may ship without a digest. Start there, run it against your last twenty merged agent patches, and count how many would have been flagged. That number is the only argument for or against this that matters.