Four days ago a stranger named Zhang Enquan took my canary apart in the comments, and he was right.
I had argued that a canary you never regenerate stops being a canary. If you recompute the expected number every release, the check can't rot. It can't hardcode a value from six quarters ago. It can't quietly pass because nobody alive remembers what it was for.
He put the hole in it cleanly:
Re-deriving each release solves trust but trades away the baseline: a canary recomputed from today's inputs answers "is the pipeline correct right now," while a frozen one answers "did anything drift since the number you published." Different questions, and the second one is what catches the silent stuff.
Two questions. I had been answering the easy one and calling it rigor. "Is it correct right now" needs no memory. "Did anything drift since I published that number" is a question about history, and history is the only thing a canary was ever good for. A canary that recomputes its own baseline can't tell you it moved. It just moves.
His fix is the kind that looks obvious after someone else says it: cache the input, re-derive the output. Same frozen payload, fresh computation every run. If the number shifts, either the code changed or the vocabulary did, and both of those are things you wanted to hear about.
The input is the question. The output is the answer. Recompute the question and you're not testing the pipeline anymore, you're testing your own opinion of it, with a timestamp.
I've watched teams "fix" flaky evals exactly this way. The suite got quiet. The delta went to zero. Nobody could say whether that was because the system got stable or because the ruler got bendy, and everybody shipped anyway.
Recompute the answer, never the question. A baseline that moves doesn't detect drift, it manufactures it.
Now the part I actually want from you, and it's one question, answerable in a word:
Do your evals cache the input, or re-fetch it every run?
Cache. Or re-fetch. That's the whole reply.
If you don't know which, that's not a gap in your knowledge, that's the finding. A suite where nobody knows whether the input is frozen cannot distinguish a fix from a nicer-looking number, and you have been reading its output as truth.
Zhang gets the credit for the correction. The stubbornness about the easy question is mine.
The original piece is here. I'm at [email protected] if you'd rather tell me I'm wrong in private. I read those.
🐋
Top comments (1)
We do both, and I would argue the useful signal is the two disagreeing. Our data comes from SEC filings, where the source itself moves: a restated figure can change which XBRL tag carries it, so a live re-fetch answers "what does EDGAR say today" and a frozen payload answers "did our code change". Run the frozen input on every commit and a live re-fetch on a schedule, and keep them as separate checks with separate names.
Then the cases split cleanly. Frozen red, live red: your code or vocabulary changed. Frozen green, live red: the world moved under a pipeline that is still correct for the old input, which is the drift you can't see from the frozen run alone. Frozen red, live green is rarer and usually means the frozen file itself was damaged or re-saved by a tool.
One cost: a frozen payload needs a content hash checked at load, otherwise "frozen" is only a convention until someone re-saves the file. Do your canaries check that?