The fine-tune looked better because the evaluation was broken

One early fine-tune appeared to beat its baseline. The result disappeared when the baseline and fixture were inspected.

The baseline could not be reproduced, and the first fixture set measured the wrong behavior. The improvement was a property of the ruler, not the candidate.

That failure became the evaluation discipline for the rest of PostTrainLLM. Baselines are frozen before training. Exact match and execution accuracy remain separate metrics. Dataset overlap and contamination are checked. Protected slices make regressions visible. Required gates fail closed rather than converting missing evidence into a zero or a pass.

The practical lesson is uncomfortable: model training can be working exactly as implemented while the experiment is still invalid. A clean loss curve does not repair an ungrounded fixture. A candidate score does not mean much when the baseline came from another binary, prompt, or dataset revision.

The factory now treats evaluation provenance as part of the model artifact. That creates more rejected results, which is better than manufacturing progress.

The full account is at https://posttrainllm.com/devlog.