Treat a Model Upgrade Like a Schema Migration

The most common LLMOps incident I've seen is not a prompt bug or a jailbreak. It's a one-line diff.

Someone bumps gpt-4o to gpt-4o-2024-08-06 (or claude-3-5-sonnet-latest to a newer snapshot), CI passes because nothing asserts on model output shape, and three days later a downstream consumer crashes on priority: null where it used to get "normal". Nobody changed the prompt. Nobody changed the code. The provider changed the weights, or the alias started pointing somewhere else.

If you come from backend work, you already have the discipline for this: you don't swap a column type in production in one deploy. You expand, migrate, contract. Apply that to models.

The version string is a pointer, not a contract

Most providers give you two things: a floating alias (-latest, or a bare family name) and a dated snapshot. The alias is a pointer that moves. The snapshot is a pin that stays put until the provider retires it — and retirements are announced, scheduled, and then they happen whether or not you upgraded.

So the pin belongs in config, not scattered across call sites:

• One registry mapping a logical task name (extract-ticket, summarize-thread) to a dated snapshot.
• The registry versioned in git, so a model change is a reviewed diff with a revert button.
• Prompt hash and output schema version in the same record. If you can't express "model X + prompt hash Y + schema v2" as a single deployable tuple, you don't have a rollback story.

The floating alias is fine in a notebook. In production it means a third party can ship a behavior change to you on a Tuesday.

Structured output is the schema, and it drifts

If your call site does json.loads(resp.content) and hands the dict downstream, the model's output is your API contract, with all the usual problems plus a few extra:

• Enums grow. A new snapshot adds "urgent" to a field you validate as low|normal|high. Strict schema validation turns that into a hard failure; lenient validation turns it into the wrong branch.
• Nulls move. null, "null", "N/A", or a missing key. All four mean "absent", and each one breaks a different consumer.
• Numbers become strings. "3" instead of 3. Pydantic in lax mode coerces it, which hides the shift until something concatenates.
• Refusals are valid. A safety refusal is a 200 with prose. Your parser fails, your retry loop retries, and you pay for three refusals.
• Truncation is silent unless you check finish_reason. A response cut off at max_tokens can still be near-valid. Always assert finish_reason == "stop" before parsing, and treat "length" as an error, not a retry-forever.

The fix is the same expand/contract you'd use for a JSON API. During the expand phase, the reader accepts both shapes and logs which one it saw. Once the logs show the old shape is gone, you delete the branch.

# pins.py — the model version lives in exactly one place, and readers are tolerant.
from dataclasses import dataclass
import hashlib, json, logging

log = logging.getLogger(__name__)

@dataclass(frozen=True)
class ModelPin:
    task: str            # logical name, e.g. "extract-ticket"
    model: str           # dated snapshot; never "-latest"
    schema_version: int
    prompt_sha: str

PINS = {
    "extract-ticket": ModelPin("extract-ticket", "gpt-4o-2024-08-06", 2, "9f3c1a"),
}

def cache_key(pin: ModelPin, user_input: str) -> str:
    # Cache on the pin, not just the input. Otherwise a rollback serves
    # answers produced by the model you just rolled back from.
    raw = f"{pin.model}|{pin.schema_version}|{pin.prompt_sha}|{user_input}"
    return hashlib.sha256(raw.encode()).hexdigest()

def read_ticket(resp_content: str, finish_reason: str) -> dict:
    if finish_reason != "stop":
        raise ValueError(f"non-stop finish: {finish_reason}")  # "length" == truncated
    payload = json.loads(resp_content)
    # Expand phase: accept v1 and v2, log which one showed up, contract later.
    if payload.get("schema_version") == 2:
        log.info("ticket_schema_seen", extra={"shape": "v2"})
        return {"title": payload["summary"], "priority": payload["priority"]["level"]}
    log.info("ticket_schema_seen", extra={"shape": "v1"})
    return {"title": payload["title"], "priority": payload.get("priority", "normal")}

Ship the tolerant reader before you change the pin. That ordering is the whole trick, and it's the step people skip on Friday afternoons.

Embeddings: never mix generations in one index

This one is nastier than output drift because it fails silently and partially. Embedding models change dimensionality, normalization, and neighborhood structure between versions. Vectors from two generations are not comparable, even if the dimensions happen to match. Cosine similarity still returns a number, so your search still returns results — just wrong ones, with no error.

Version the index in its name: docs_v2_<model>_<dims>. Write new documents to both old and new during the backfill, flip reads when the backfill completes, then drop the old collection. Budget for the full re-embedding cost up front; for a large corpus that's the dominant cost of the upgrade, not the chat tokens.

Shadow traffic: measure the right thing

Before the pin moves for real, dual-run. Send a sample of live (or replayed) inputs to the candidate and score both outputs. Two things matter here:

• Sample size. Small samples tell you nothing about a 2% change in failure rate. Decide the minimum sample before you look, or you'll stop at the first run that looks good.
• What you measure. Per-call latency is misleading. Measure tokens per resolved task, because newer models often get more verbose — a cheaper per-token price with 40% more output tokens and more retries can be a net loss. Track task success, retry rate, output token ratio, and schema-violation rate together.

If you use an LLM judge, calibrate it against a hand-labeled set first and report its agreement rate. An uncalibrated judge is a random number generator with good manners.

Rollback is not undo

Flipping the pin back does not un-send the email, un-issue the refund, or un-create the Jira ticket. Model changes are usually paired with tool calls, and tool calls have side effects.

Before you allow a pin change in a write path:

• Idempotency keys on every side-effecting tool call, keyed on the logical operation, not the attempt.
• An outbox or dry-run mode you can route a candidate model into, so shadow traffic never writes.
• A kill switch that stops new work, separate from the pin, so you can freeze the queue while you decide.

Rolling back the model and rolling back the effects are two different runbooks. Write both.

Pitfalls

• Caching on input only. After a rollback, you serve the old model's cached answers and conclude the rollback didn't work.
• Retrying on parse failures. Refusals and truncations are deterministic-ish; three retries triple the bill and change nothing.
• Contracting too early. Dropping the v1 reader the day the new pin ships guarantees an incident on the first straggler.
• Testing against live providers in CI. Flaky, slow, and it re-introduces the floating alias you just removed. Record fixtures from the pinned snapshot instead.
• Only measuring the happy path. The interesting deltas are in refusal rate, schema violations, and tool-argument validity.

Monday checklist

• [ ] Every model call resolves through a single registry with dated snapshots. No -latest outside notebooks.
• [ ] The registry records model, prompt hash, and schema version together, and it's in git.
• [ ] finish_reason is asserted before parsing anywhere output is consumed.
• [ ] Output readers accept the previous schema version and log which shape they saw.
• [ ] Cache keys include the model pin and schema version.
• [ ] Embedding collections are named with model and dimension; re-embedding cost is estimated.
• [ ] Shadow traffic runs on a pre-declared sample size, with tokens-per-task and schema-violation rate in the report.
• [ ] Every side-effecting tool call has an idempotency key and a dry-run path.
• [ ] You have a written rollback runbook with two sections: revert the pin, repair the effects.
• [ ] CI runs contract tests against recorded fixtures from the pinned snapshot, not the live API.