Nothing happened. That is the strange part.
No deploy. No pull request. Nobody touched the prompt. Your agent ran the way it always ran on Friday,...
For further actions, you may consider blocking this person and/or reporting abuse
The "tool choice reaches the bill before it reaches anyone's attention" line is painfully accurate. In our experience that's actually the easiest of the three to catch, precisely because it's metered — a step-count or tool-call histogram per task, tracked over time, throws a visible bump the day a model's first move changes, even when quality looks fine. We alert on "calls per completed task" drifting outside a band, and it's caught two provider-side changes we'd otherwise have missed.
The frozen baseline is the real answer, and I'd add one refinement: build the baseline out of your ambiguous requests, not the clean ones. Underspecified inputs are exactly where house-style shifts show up, and they're the questions people forget to put in eval sets because they're annoying to grade. A rubric that scores "did it ask a clarifying question vs. assume" on a fixed set of vague prompts has been our most sensitive tripwire for the ambiguity drift you describe. How are you handling grading on those — LLM-judge, or human spot-checks?
The ambiguous-request point is excellent. I agree that the “annoying to grade” cases are probably some of the most valuable ones to freeze, because they expose behavioral changes that clean benchmark prompts can completely miss.
For grading, I’d use a combination rather than relying entirely on an LLM judge. For something like “did it clarify or assume?”, the expected behavior can be made fairly explicit, so I’d prefer a deterministic check where possible, with human review for the cases where the rubric itself is ambiguous.
I’m also wary of using one LLM to judge another model’s behavior without a fixed rubric and some human calibration. Otherwise you can end up measuring agreement between models rather than whether the behavior actually meets the requirement.
And I really like your “calls per completed task” metric. That’s exactly the kind of operational signal that can reveal a behavioral regression before someone notices a quality problem.
The frozen baseline has a second expiry nobody schedules: the model is not the only thing that moves, the traffic is. Freeze real requests in March and by September some of them ask about a flow that no longer exists, so the diff comes back clean because the set stopped representing what people actually send. How are you deciding when a baseline set has aged out, separate from when the model changes?
That’s a really good point. I wouldn’t give the baseline a fixed calendar expiry, because six months can be irrelevant in one system and perfectly valid in another.
I’d watch for changes in the request distribution and the underlying workflow: new task types, changing frequencies, retired flows, or shifts in ambiguity patterns. Those are signals that the baseline is no longer representative.
I’d also keep a stable core of historically important cases while refreshing a portion of the set from recent real traffic. That gives you continuity for regression detection without letting the baseline become a museum piece.
A frozen baseline needs a noise floor next to it, or the first diff after an upgrade is unreadable. Running the same set twice against the model you are on now, before anything changes, gives you the rate at which outputs move on their own; anything under that rate is sampling, not the new model. It also helps to store the resolved model ID with each recorded output rather than the alias you called, since an alias can point at a different snapshot later and then you cannot tell which version produced the baseline.
Yes, the noise floor is an important addition. Otherwise we can end up treating normal output variance as a regression, especially with stochastic models.
And I strongly agree on storing the resolved model ID rather than only the alias. A baseline is only useful if we can actually reproduce the conditions under which it was created.
I’d go one step further and think of the baseline as a record of the whole evaluation condition, not just the model output: model version, configuration, inputs, expected behavior, and the resulting evidence. Otherwise six months later we may know that something changed without being able to establish what actually changed.
On the record-the-whole-condition part, one thing I would add: build the record from a read-back on the same surface the comparison will read from, rather than from the values you sent. I hit this on an ordinary REST API this week, where a field I had just written was simply absent from the response of the endpoint you would naturally check it on, so
.get()returnedNoneand my check read that as unset. I now assert that the key is present rather than that its value is non-null, because otherwise "never configured" and "this surface does not expose it" end up as the same row in the baseline, and months later those are not the same thing.That’s a great distinction. Recording what we sent versus what the system actually exposes can quietly create two different baselines without anyone noticing.
I especially like the “absent vs null” point. They may look equivalent to a simple check, but they represent completely different system states. If the baseline is supposed to be evidence we can rely on months later, those states need to remain distinguishable.
The frozen baseline idea is the part that matches day to day agent work. The failure mode that bites hardest is not worse answers. It is a quiet shift in first tool choice, where the agent stops reaching for a search tool it used to call and answers from stale context instead. Shape checks and JSON schema tests stay green, because the response still parses. Replaying a handful of real underspecified tickets side by side is the only thing that makes the new filling in of gaps visible.
Yes, exactly. That distinction between “the output still parses” and “the agent is still behaving correctly” is the dangerous part.
A schema test can tell us that the response has the right shape. It can't tell us that the agent made the right decision about whether to search, use a tool, ask for clarification, or rely on existing context.
That is why I like the underspecified tickets as regression cases. They test the decision boundary, not just the final output.
A model upgrade can leave all the obvious checks green while quietly changing the agent's policy. Those are the regressions I think are easiest to miss and hardest to explain after the fact.
The "better on average" point is exactly what bit me. A provider bumped a model, aggregate scores looked fine, but a few behaviors I'd quietly encoded in prompts shifted and nothing flagged it. I now pin versions and run a small behavioral suite on every upgrade, how do you catch these before prod?
That’s exactly why I like keeping the behavioral suite small and intentional. I’d run it against every model change before allowing the new version into the production path.
The key for me is that the suite shouldn’t only measure aggregate quality. I’d freeze the behaviors that matter to the application: tool choice, clarification vs. assumption, output constraints, safety boundaries, and other known failure modes.
Then compare the new run against the previous baseline at the individual behavior level. An improvement in the average score shouldn’t be able to hide a regression in a behavior we explicitly care about.
That gives you a much better signal before production than relying on the provider’s benchmark or an overall quality score.
the eval set will not catch it if your eval set is made of clear, well formed questions is the one that hurts.
hit this in prod about 4 months ago. Anthropic bumped a model mid sprint, our eval suite stayed green, and then support started flagging that the agent's tool selection had quietly shifted. it was defaulting to memory retrieval instead of a specialized search tool we'd built. billable tool calls dropped 40%, but answer quality degraded on the edge cases our evals happened to skip.
we now snapshot real production requests weekly. messy ones, not the cleaned up versions. manual judgment on the delta is unavoidable.
what format do you keep your frozen baselines in — raw request/response pairs or something more structured?
I’d keep both, but make the structured record the baseline and preserve the raw request/response alongside it. The raw pair is the evidence; the structured layer makes comparison and analysis possible. I’d also pin the resolved model ID, relevant configuration, tool availability, and timestamp with each run. Otherwise a “frozen” baseline can quietly become a record of inputs and outputs without preserving the conditions that produced them. And I agree on keeping the messy production requests. Those are often where the behavioral drift becomes visible first.
The ambiguity point hits different. The model confidently answering the wrong question is so much worse than one that errors out — at least an error you catch.
Exactly. An explicit failure gives you a signal. A confident answer to the wrong question can pass every superficial check because it looks perfectly reasonable.
That’s why I think ambiguity cases are so valuable in evals. They test whether the agent recognizes uncertainty and chooses the right behavior, not just whether it can produce a plausible answer.
This alone should be said louder to all software engineers. It will save you from a lot of headache in the future.
Yes. I’d phrase the distinction as: automate the execution of the evaluation, but be very careful about automating the definition of “good.”
You can automate running tests, collecting outputs, calculating deterministic checks, and flagging changes. But the criteria for acceptable behavior should come from the requirements and be calibrated with human judgment, especially for ambiguous cases.
Otherwise we risk building a very efficient machine for measuring the wrong thing.