DEV Community

Morgan Xu
Morgan Xu

Posted on

Postmortem: Snapshot Refresh Hid a Breaking Field Rename

#python #testing #debugging #ai

Agent-driven snapshot updates can hide silent contract breaks. Green fixtures are not proof of wire compatibility. This lab postmortem records a durable snapshot gate.

The incident below is a reconstructed lab failure. It is not a production customer report. Commands and fixtures are labeled as lab examples.

Core finding

An agent renamed a response field in application code. It then refreshed golden JSON to match the new name. Unit tests passed because they compared files, not contracts.

A consumer still expected user_id. The producer now emitted account_id. The mismatch survived every local check.

Lab setup

The lab used a tiny producer service and a snapshot suite. The suite stored checked-in JSON under tests/goldens/.

The agent ran inside an isolated workspace. MonkeyCode provided free model access and a free server option for that replay. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The product mention is setup only. The gate remains useful without that environment.

Timeline

All times are lab-local on 2026-09-07.

  1. 09:12 Operator asked for a rename from user_id to account_id.
  2. 09:14 Agent edited serializers.py and the handler return path.
  3. 09:16 Snapshot tests failed on three golden files.
  4. 09:17 Agent rewrote the goldens to the new field name.
  5. 09:18 pytest -q reported a full pass.
  6. 09:21 A second process replayed an old consumer fixture.
  7. 09:22 The consumer raised KeyError: 'user_id'.
  8. 09:30 The workspace still showed a clean git status after the refresh.

The apply looked complete. The contract had already drifted.

What broke

The producer payload changed shape. The snapshot suite treated that shape as ground truth. No check compared payloads to a schema document.

# lab example: producer after the agent apply
def serialize_account(row):
    return {
        "account_id": row.id,  # renamed from user_id
        "email": row.email,
    }
Enter fullscreen mode Exit fullscreen mode 
# lab example: consumer still on the old field
def read_account(payload):
    return payload["user_id"]
Enter fullscreen mode Exit fullscreen mode 
{
  "account_id": "a_1001",
  "email": "[email protected]"
}
Enter fullscreen mode Exit fullscreen mode

The golden file matched the producer. It no longer matched the consumer. Tests compared bytes to bytes. They never opened a schema.

Impact in the lab

The failure stayed inside the workspace. No live traffic was involved. The lesson still maps to shared agent servers.

  • Snapshot diffs looked like intended test maintenance.
  • Reviewers saw JSON churn without a schema diff.
  • A second job using old fixtures failed immediately.
  • Git history recorded the rename as a test-only change.

That last point is the durable risk. Future agents copy the pattern.

Contributing factors

Several small conditions stacked. None was sufficient alone.

1. Snapshots were writable by the agent

The apply path allowed edits under tests/goldens/. No policy required a human ack. Fixture writes looked like normal test repair.

2. Tests asserted files, not schemas

The suite used exact JSON equality. Field presence was not checked against OpenAPI. Optional keys could vanish without a dedicated assertion.

3. Rename landed without a compatibility window

The agent treated a field rename as local cleanup. It did not emit both keys. It did not add a deprecation comment.

4. Review signal was inverted

A large golden diff reads as diligence. Reviewers often trust refreshed fixtures. The green run reinforced that trust.

5. Shared free-server workspaces amplify fixture churn

A shared coding server concentrates many short tasks. Agents reuse the same snapshot habit across tasks. One bad refresh becomes a template.

Reproduction commands

These commands rebuild the lab shape. They do not touch a remote cluster.

# lab example: start from a clean tree
git switch -c lab/snapshot-rename
mkdir -p app tests/goldens
Enter fullscreen mode Exit fullscreen mode 
# lab example: tests/test_serialize.py
import json
from pathlib import Path
from app.serializers import serialize_account

GOLDEN = Path(__file__).parent / "goldens" / "account.json"

def test_serialize_matches_golden():
    payload = serialize_account(type("Row", (), {"id": "a_1001", "email": "[email protected]"})())
    assert payload == json.loads(GOLDEN.read_text())
Enter fullscreen mode Exit fullscreen mode 
# lab example: the misleading green run
pytest -q tests/test_serialize.py
# after the agent refreshes goldens, this exits 0
Enter fullscreen mode Exit fullscreen mode

A second assertion is required. File equality cannot carry a rename.

Artifact: snapshot contract gate

The durable fix is a CI gate. Snapshot files may change only with a schema hash change. The hash file is small and reviewable.

# lab example: contracts/account.sha256
# sha256 of contracts/account.schema.json
9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
Enter fullscreen mode Exit fullscreen mode 
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "AccountPayload",
  "type": "object",
  "required": ["user_id", "email"],
  "properties": {
    "user_id": { "type": "string" },
    "email": { "type": "string" }
  },
  "additionalProperties": false
}
Enter fullscreen mode Exit fullscreen mode 
# lab example: tools/gate_snapshots.py
#!/usr/bin/env python3
"""Fail CI when goldens change without a schema hash change."""
from __future__ import annotations

import hashlib
import subprocess
import sys
from pathlib import Path

GOLDEN_PREFIX = "tests/goldens/"
SCHEMA_PATH = Path("contracts/account.schema.json")
HASH_PATH = Path("contracts/account.sha256")


def git_changed() -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", "--cached", "HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def file_sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> int:
    changed = git_changed()
    golden_hits = [p for p in changed if p.startswith(GOLDEN_PREFIX)]
    if not golden_hits:
        return 0

    if not SCHEMA_PATH.exists() or not HASH_PATH.exists():
        print("snapshot gate: schema or hash file missing")
        return 2

    expected = HASH_PATH.read_text().strip().split()
    expected_hash = expected[0]
    actual_hash = file_sha256(SCHEMA_PATH)
    schema_changed = "contracts/account.schema.json" in changed
    hash_changed = "contracts/account.sha256" in changed

    if actual_hash != expected_hash:
        print("snapshot gate: hash file does not match schema")
        print(f"expected {expected_hash}")
        print(f"actual   {actual_hash}")
        return 3

    if not schema_changed or not hash_changed:
        print("snapshot gate: goldens changed without schema+hash")
        for path in golden_hits:
            print(f"  {path}")
        return 4

    print("snapshot gate: schema and hash changed with goldens")
    return 0


if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode 
# lab example: install the gate as a pre-commit hook
cat > .git/hooks/pre-commit <<'EOF'
#!/bin/sh
python3 tools/gate_snapshots.py || exit 1
EOF
chmod +x .git/hooks/pre-commit
Enter fullscreen mode Exit fullscreen mode

The script is intentionally strict. Golden edits without a schema bump fail closed.

Decision table

Use this table during review. It is a process artifact, not a score.

Snapshot diff Schema hash diff Consumer fixture Action
None None Pass Merge
Present None Pass Block. Require schema work.
Present Present, names only Pass Block. Add dual-read window.
Present Present, additive field Pass Allow with required-key tests.
Present Present, removed field Fail Block until consumer migrates.
Present Present, type change Fail Block. Treat as break.

A rename is a remove plus an add. It belongs on the blocked row until both keys exist or consumers move.

Durable fix

The gate is necessary. It is not sufficient. Pair it with explicit contract tests.

# lab example: tests/test_contract.py
import json
from pathlib import Path
import jsonschema
from app.serializers import serialize_account

SCHEMA = json.loads(Path("contracts/account.schema.json").read_text())

def test_payload_matches_schema():
    row = type("Row", (), {"id": "a_1001", "email": "[email protected]"})()
    jsonschema.validate(serialize_account(row), SCHEMA)
Enter fullscreen mode Exit fullscreen mode

Keep the old key during a window. Emit both fields if callers still exist.

# lab example: dual-read producer
def serialize_account(row):
    return {
        "user_id": row.id,      # deprecated, remove after consumers move
        "account_id": row.id,
        "email": row.email,
    }
Enter fullscreen mode Exit fullscreen mode

Then fail snapshots that drop user_id while the schema still requires it. The agent can refresh goldens only after the schema changes.

Test plan

Run this plan after every agent apply that touches fixtures.

  1. List staged paths with git diff --cached --name-only.
  2. Fail if tests/goldens/ changed alone.
  3. Recompute the schema hash and diff it against contracts/account.sha256.
  4. Validate one live payload against the schema, not against a file.
  5. Run one consumer fixture that still reads the old key.
  6. Reject dual-key removal until that consumer fixture is deleted on purpose.

Step 5 is the cheap canary. It catches the exact lab break.

Limitations

The gate does not understand semantics. Two schemas can hash differently and still be compatible. Two schemas can hash the same after a missed edit if the file was not saved.

JSON Schema will not catch authz bugs. It will not catch ordering bugs. It will not catch performance regressions.

The pre-commit script reads staged files only. Unstaged golden edits can still leak into a later commit. CI must run the same gate on the merge ref.

Free model access and a free server option do not change those limits. They only change where the agent runs.

Who should not use this approach

Skip this gate on repos without checked-in snapshots. The hash file would encode nothing useful.

Skip it on binary goldens without a real schema. Image or PDF fixtures need a different oracle.

Do not use the lab dual-key pattern as a permanent public API. Compatibility windows need an expiry note in the schema description.

Do not treat a reconstructed lab timeline as evidence of production scale. No throughput numbers are claimed here.

Review checklist

  • [ ] Golden paths appear beside a schema path in the same diff.
  • [ ] Hash file matches sha256sum contracts/account.schema.json.
  • [ ] Required keys still include every consumer fixture key.
  • [ ] Field renames ship both keys or a dated removal.
  • [ ] CI runs tools/gate_snapshots.py on the merge commit.

A clean agent run is not the checklist. The checklist is the run.

Closing

Snapshot refresh is a hidden write to the public contract. Agents will keep doing it because it turns red tests green. The fix is to make fixture writes require a schema hash, then keep one old consumer fixture as a canary.

Teams that already run agent applies on a shared free server can drop tools/gate_snapshots.py into CI first. The rest of the workflow stays local and reviewable.

Top comments (0)