The Prompt Changed, the Schema Didn't, and the Job Still Passed

Version skew is the quietest bug class in model pipelines. Validation passes because the schema never moved. Your prompt did.

That gap is where I lost a week. Here is the walkthrough, plus the harness I now use to catch it.

The symptom: green checks, wrong totals

The extraction job ran nightly. It pulled invoice lines and wrote normalized amounts.

One morning the totals were low. No exceptions. No retries. No dead letters.

The validator was happy. Every record had amount as a string. That was the only rule.

So what did the tests actually cover? Type shape. Nothing else. Not format, not provenance, not drift.

Root cause: one sentence in the prompt

I had edited the prompt the week before. I added a line asking the model to keep amounts human-readable.

So it started returning 1,200.00 instead of 1200.00. Both are strings. Both passed.

Downstream, Decimal("1,200.00") raised. The wrapper caught it and returned None.

None summed to zero. The nightly report still rendered.

Three separate defects lined up: a loose schema, a swallowed exception, and no record of which prompt produced which output.

Fix any one of them and the bug is loud. Fix none, and it is silent for as long as nobody compares.

Step 1 — Freeze the input set first

Before touching a prompt, capture a fixed fixture file. Same inputs, forever.

Twenty to fifty records is enough. Include the ugly ones: multi-currency, credits, blank lines.

mkdir -p fixtures runs
python -m harness.snapshot --source prod-sample --limit 50 \
  --out fixtures/invoice-50.jsonl

Never regenerate this file during debugging. It is your ground truth for comparison.

Step 2 — Stamp provenance into every record

This is the part I skipped, and it cost me the most time. Every output needs to say what produced it.

import hashlib, json, pathlib

PROMPT = pathlib.Path("prompts/extract.v3.txt").read_text()
SCHEMA = pathlib.Path("schemas/invoice.json").read_text()

def fp(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:12]

PROVENANCE = {
    "prompt_fp": fp(PROMPT),
    "schema_fp": fp(SCHEMA),
}

def record(input_id: str, payload: dict) -> dict:
    return {
        "input_id": input_id,
        "meta": PROVENANCE,
        "output": payload,
    }

Twelve hex characters is plenty. You are detecting change, not defending against attack.

Write these fields to JSONL, not to logs. Logs rotate. Debugging artifacts should not.

Step 3 — Reproduce in a clean environment

Half my confusion was local state. A cached prompt. A stale schemas/ copy. An editable install.

So I reran the whole job in a fresh box with nothing cached.

That is where MonkeyCode's free model access and free server option became relevant to the method. I used a hosted, disposable environment for the clean run, and free model access for the replay passes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Check the project's current terms yourself before you depend on them. I am not quoting quotas or specs here, because those change.

python -m harness.replay --input fixtures/invoice-50.jsonl \
  --out runs/clean.jsonl --no-cache

The rule is simple. The repro must not share a filesystem with your dev shell.

Step 4 — Diff by fingerprint, and refuse mixed runs

Now the useful part. Compare old output to new output, grouped by provenance.

from collections import defaultdict

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

def diff(baseline_path, candidate_path, allow_skew=False):
    base = {(r["input_id"]): r for r in load(baseline_path)}
    cand = {(r["input_id"]): r for r in load(candidate_path)}

    changed = defaultdict(list)
    for key, new in cand.items():
        old = base.get(key)
        if old is None:
            changed["missing_baseline"].append(key)
            continue
        if not allow_skew and old["meta"] != new["meta"]:
            changed["skewed"].append(key)
            continue
        if old["output"] != new["output"]:
            changed["output_changed"].append(key)
    return changed

The skewed bucket is the whole point. Without --allow-skew, the tool refuses to compare two runs from different prompts.

That refusal is a feature. It tells you the question you asked was malformed.

Run it:

python -m harness.diff runs/baseline.jsonl runs/clean.jsonl

If skewed is non-empty and you expected a clean comparison, stop. You have two variables, not one.

Step 5 — Constrain the format, not just the type

A schema that says string where you mean decimal is a wish, not a contract.

{
  "amount": {
    "type": "string",
    "pattern": "^-?[0-9]+(\\.[0-9]{2})?$"
  },
  "currency": {
    "type": "string",
    "enum": ["USD", "EUR", "GBP"]
  }
}

Now 1,200.00 fails at the boundary, loudly. The prompt can still be friendly. The contract stays strict.

And stop swallowing parse errors. A missing amount should be a dead letter, not a zero.

Step 6 — Add the guard test

One test would have caught this in minutes. It costs four lines.

def test_every_record_has_current_provenance():
    rows = load("runs/clean.jsonl")
    assert rows, "empty run"
    for row in rows:
        assert row["meta"]["prompt_fp"] == PROVENANCE["prompt_fp"]
        assert row["meta"]["schema_fp"] == PROVENANCE["schema_fp"]

Also assert the fixture count. A silent zero-row run is its own incident.

When a hosted free run is enough

Not every debugging task needs this ceremony. Here is how I decide.

Two rules from that table. Never send production records to an environment you do not control. And never trust a timing number from a machine you did not size.

Limitations, and who should skip this

The harness is small on purpose. It has no retries, no rate limiting, and no backoff.

It assumes your outputs are JSON-serializable and order-stable. Flaky ordering will show up as output_changed noise.

Skip this if you have a single prompt, a stable schema, and no downstream consumers. The overhead is not worth it.

Skip it too if you cannot hold a representative fixture set. A skewed diff over bad inputs is just more noise.

If you want to run the clean-environment step without provisioning anything, the project's free model access and free server option are one way in. The harness does not care which provider you use.

The habit worth keeping

Stamp provenance before you change a prompt. Diff before you trust a total.

When a check goes green, ask what it actually asserted. Type shape is rarely the thing you meant.

Run the diff before you run the fix. That order saves a week.