Nothing happened. That is the strange part.
No deploy. No pull request. Nobody touched the prompt. Your agent ran the way it always ran on Friday, and it runs on Monday, and every dashboard is green. Then a ticket comes in about an answer nobody on your team would have written, and you go looking for the change that caused it, and there is no change on your side. There was a model upgrade.
It is the only change to your system that you did not make, cannot find in your own git history, and usually cannot roll back on your own schedule. It is also the one most likely to be announced to you as good news.
Why a model upgrade does not look like a bug
Because it is not one. The new model really is better. Better on reasoning, better on code, better on the evaluations the lab published beside it, and probably better on yours too, if what you measured was the average.
Better and same are different words. Your product was not built on the average. It was built on a specific set of behaviours you watched, liked, and then quietly encoded into everything downstream: how long the answers run, how much the thing hedges, which tool it reaches for first, what it does when a request is vague. None of that appears in release notes. All of it can move.
And when it moves, nothing throws. There is no stack trace for "this answer is now worse in a way a customer will notice." Your tests keep passing, because your tests check that the JSON parses and the fields are there, and the JSON still parses and the fields are still there.
The three things that actually move
Shape. Answers get longer, or shorter, or start opening with a summary they never used to open with. Harmless, right up until something downstream was written against the old shape.
Tool choice. The agent develops a new favourite first move. It takes six calls to do what used to take three, or it stops calling the tool you built for it because it has decided it can answer from memory. This one usually reaches the bill before it reaches anyone's attention.
Ambiguity. This is the expensive one. Most real requests are underspecified, and every model has a house style for filling in the gap. When that style changes, your agent starts confidently answering a slightly different question than the one it used to answer. Your eval set will not catch it if your eval set is made of clear, well-formed questions, and most eval sets are, because clear questions are the easy ones to write.
What catches it
One thing, and it is boring. A frozen baseline.
Take a set of real requests. Not invented ones, not the ones you wish people sent. Run them against the model you are on right now and keep the outputs, together with your own verdict on each one, written while you still have the old behaviour in front of you. That file is the only thing standing between you and hearing about it from a customer.
When the upgrade lands, run the same set again and put the two side by side. What you get is a diff, and here is the honest limit: a diff does not tell you which side is better. It tells you what moved. A person still has to read the ones that changed and decide whether each change is an improvement or a regression, and that reading is the actual work. Anthropic's own writeup on evaluating agents lands in the same place. Automate the running. Do not try to automate the judging.
Build it before you need it. If you start once the upgrade is already live, your baseline is contaminated by the thing you are trying to measure, and you will lose a week arguing about whether the agent used to do that.
This is the normal condition now, not an event
The model under you will keep changing. That is not an occasional disruption to plan around, it is the ground you are building on from here.
Which is why the interesting skill stopped being prompt work a while ago. It is the machinery around the model: the frozen cases, the recorded verdicts, the diff you can run in an afternoon, the decision about what a failure actually costs you. That has a name now. It is called harness engineering, and this is the exact situation it exists for.
You do not need a platform to start. Twenty real requests, the answers you get today, and your honest opinion of each one, written down before anything changes. Do that this afternoon and the next model upgrade is an inconvenience instead of a surprise.
Top comments (18)
The "tool choice reaches the bill before it reaches anyone's attention" line is painfully accurate. In our experience that's actually the easiest of the three to catch, precisely because it's metered — a step-count or tool-call histogram per task, tracked over time, throws a visible bump the day a model's first move changes, even when quality looks fine. We alert on "calls per completed task" drifting outside a band, and it's caught two provider-side changes we'd otherwise have missed.
The frozen baseline is the real answer, and I'd add one refinement: build the baseline out of your ambiguous requests, not the clean ones. Underspecified inputs are exactly where house-style shifts show up, and they're the questions people forget to put in eval sets because they're annoying to grade. A rubric that scores "did it ask a clarifying question vs. assume" on a fixed set of vague prompts has been our most sensitive tripwire for the ambiguity drift you describe. How are you handling grading on those — LLM-judge, or human spot-checks?
The ambiguous-request point is excellent. I agree that the “annoying to grade” cases are probably some of the most valuable ones to freeze, because they expose behavioral changes that clean benchmark prompts can completely miss.
For grading, I’d use a combination rather than relying entirely on an LLM judge. For something like “did it clarify or assume?”, the expected behavior can be made fairly explicit, so I’d prefer a deterministic check where possible, with human review for the cases where the rubric itself is ambiguous.
I’m also wary of using one LLM to judge another model’s behavior without a fixed rubric and some human calibration. Otherwise you can end up measuring agreement between models rather than whether the behavior actually meets the requirement.
And I really like your “calls per completed task” metric. That’s exactly the kind of operational signal that can reveal a behavioral regression before someone notices a quality problem.
The frozen baseline has a second expiry nobody schedules: the model is not the only thing that moves, the traffic is. Freeze real requests in March and by September some of them ask about a flow that no longer exists, so the diff comes back clean because the set stopped representing what people actually send. How are you deciding when a baseline set has aged out, separate from when the model changes?
That’s a really good point. I wouldn’t give the baseline a fixed calendar expiry, because six months can be irrelevant in one system and perfectly valid in another.
I’d watch for changes in the request distribution and the underlying workflow: new task types, changing frequencies, retired flows, or shifts in ambiguity patterns. Those are signals that the baseline is no longer representative.
I’d also keep a stable core of historically important cases while refreshing a portion of the set from recent real traffic. That gives you continuity for regression detection without letting the baseline become a museum piece.
A frozen baseline needs a noise floor next to it, or the first diff after an upgrade is unreadable. Running the same set twice against the model you are on now, before anything changes, gives you the rate at which outputs move on their own; anything under that rate is sampling, not the new model. It also helps to store the resolved model ID with each recorded output rather than the alias you called, since an alias can point at a different snapshot later and then you cannot tell which version produced the baseline.
Yes, the noise floor is an important addition. Otherwise we can end up treating normal output variance as a regression, especially with stochastic models.
And I strongly agree on storing the resolved model ID rather than only the alias. A baseline is only useful if we can actually reproduce the conditions under which it was created.
I’d go one step further and think of the baseline as a record of the whole evaluation condition, not just the model output: model version, configuration, inputs, expected behavior, and the resulting evidence. Otherwise six months later we may know that something changed without being able to establish what actually changed.
On the record-the-whole-condition part, one thing I would add: build the record from a read-back on the same surface the comparison will read from, rather than from the values you sent. I hit this on an ordinary REST API this week, where a field I had just written was simply absent from the response of the endpoint you would naturally check it on, so
.get()returnedNoneand my check read that as unset. I now assert that the key is present rather than that its value is non-null, because otherwise "never configured" and "this surface does not expose it" end up as the same row in the baseline, and months later those are not the same thing.That’s a great distinction. Recording what we sent versus what the system actually exposes can quietly create two different baselines without anyone noticing.
I especially like the “absent vs null” point. They may look equivalent to a simple check, but they represent completely different system states. If the baseline is supposed to be evidence we can rely on months later, those states need to remain distinguishable.
The frozen baseline idea is the part that matches day to day agent work. The failure mode that bites hardest is not worse answers. It is a quiet shift in first tool choice, where the agent stops reaching for a search tool it used to call and answers from stale context instead. Shape checks and JSON schema tests stay green, because the response still parses. Replaying a handful of real underspecified tickets side by side is the only thing that makes the new filling in of gaps visible.
Yes, exactly. That distinction between “the output still parses” and “the agent is still behaving correctly” is the dangerous part.
A schema test can tell us that the response has the right shape. It can't tell us that the agent made the right decision about whether to search, use a tool, ask for clarification, or rely on existing context.
That is why I like the underspecified tickets as regression cases. They test the decision boundary, not just the final output.
A model upgrade can leave all the obvious checks green while quietly changing the agent's policy. Those are the regressions I think are easiest to miss and hardest to explain after the fact.
The "better on average" point is exactly what bit me. A provider bumped a model, aggregate scores looked fine, but a few behaviors I'd quietly encoded in prompts shifted and nothing flagged it. I now pin versions and run a small behavioral suite on every upgrade, how do you catch these before prod?
That’s exactly why I like keeping the behavioral suite small and intentional. I’d run it against every model change before allowing the new version into the production path.
The key for me is that the suite shouldn’t only measure aggregate quality. I’d freeze the behaviors that matter to the application: tool choice, clarification vs. assumption, output constraints, safety boundaries, and other known failure modes.
Then compare the new run against the previous baseline at the individual behavior level. An improvement in the average score shouldn’t be able to hide a regression in a behavior we explicitly care about.
That gives you a much better signal before production than relying on the provider’s benchmark or an overall quality score.
the eval set will not catch it if your eval set is made of clear, well formed questions is the one that hurts.
hit this in prod about 4 months ago. Anthropic bumped a model mid sprint, our eval suite stayed green, and then support started flagging that the agent's tool selection had quietly shifted. it was defaulting to memory retrieval instead of a specialized search tool we'd built. billable tool calls dropped 40%, but answer quality degraded on the edge cases our evals happened to skip.
we now snapshot real production requests weekly. messy ones, not the cleaned up versions. manual judgment on the delta is unavoidable.
what format do you keep your frozen baselines in — raw request/response pairs or something more structured?
I’d keep both, but make the structured record the baseline and preserve the raw request/response alongside it. The raw pair is the evidence; the structured layer makes comparison and analysis possible. I’d also pin the resolved model ID, relevant configuration, tool availability, and timestamp with each run. Otherwise a “frozen” baseline can quietly become a record of inputs and outputs without preserving the conditions that produced them. And I agree on keeping the messy production requests. Those are often where the behavioral drift becomes visible first.
The ambiguity point hits different. The model confidently answering the wrong question is so much worse than one that errors out — at least an error you catch.
Exactly. An explicit failure gives you a signal. A confident answer to the wrong question can pass every superficial check because it looks perfectly reasonable.
That’s why I think ambiguity cases are so valuable in evals. They test whether the agent recognizes uncertainty and chooses the right behavior, not just whether it can produce a plausible answer.
This alone should be said louder to all software engineers. It will save you from a lot of headache in the future.
Yes. I’d phrase the distinction as: automate the execution of the evaluation, but be very careful about automating the definition of “good.”
You can automate running tests, collecting outputs, calculating deterministic checks, and flagging changes. But the criteria for acceptable behavior should come from the requirements and be calibrated with human judgment, especially for ambiguous cases.
Otherwise we risk building a very efficient machine for measuring the wrong thing.