Five ways my own test harness lied to me, and the one that nearly got published
The number that gave it away was sitting in a table I ha...
For further actions, you may consider blocking this person and/or reporting abuse
The 4.3 -> 7.6 tok/s reading may be a cold start rather than contention. If the speed is computed from wall-clock time, the first generations include Ollama's model load, and with short outputs that dominates. The response splits it out: load_duration, prompt_eval_duration, and eval_count / eval_duration (nanoseconds), so tokens per second from eval_duration alone leaves load and prompt processing out. Worth keeping the first call out of the "first third", or reporting it on its own.
One more on the 13 of 15 NOT IN THE MATERIAL: with n=15 the Wilson 95% interval for 13/15 is about 62% to 96%, and 4 of 4 abstentions has a lower bound near 51%. Printing the interval next to the headline would carry the same warning your concentration check does. Does the decay check read eval_duration, or time around the request?
Good catches, and checking them properly turned up something worse than either of us suggested.
First, directly: the decay check reads
eval_duration, not wall clock.tpsiseval_count / eval_duration, andload_durationonly feeds a separatefirst_token_msfield the check never reads. So cold start can't explain it. The load was real and enormous: 96.7 seconds on the first QA call, down to a 2.8-second median afterward, but it never showed up in that number.You were right that short outputs break this, just by a different mechanism. Reported tok/s fits
n / ((n-1) × t)to within 2% across my runs. That's consistent witheval_countcounting n tokens whileeval_durationcovers approximately n-1 inter-token intervals. The resulting inflation is n/(n-1): 2x at two tokens, 1.12x at nine.In the extraction run, the answers drop from nine tokens to two partway through, which explains the entire apparent increase from 4.3 to 7.6 tok/s. Inter-token time is flat. Nothing got faster.
So the metric I built specifically so that answer length wouldn't contaminate the speed reading is contaminated by answer length.
Your interval arithmetic is right too: 62.1% to 96.3% for 13/15, and a 51.0% lower bound for 4/4. These are hand-built fixtures rather than a random sample, so those intervals illustrate the uncertainty in extrapolating from small counts, not a statistically justified bound on general capability.
Thanks. This is the second measurement error in this project that someone other than me found.
The 97-median-token line is the whole post in one number: the harness reported the empty reference block correctly, three reads in a row, and it got past you because the scores came back internally consistent with plausible, different failure modes on each side — which is precisely what a successful-looking run is supposed to feel like. Every call returned 200, the scorer scored, 2,700 words of conclusions got written, and the entire edifice measured a question nobody was asking. Worth saying the wine-inventing model was not malfunctioning either: it answered the prompt it actually received, and that prompt contained nothing. The only number telling the truth was in a table you had already read — telemetry stays honest, humans skim. That asymmetry is the reusable part.
That's an excellent reading of what went wrong. The measurements were internally consistent, the model behaviors looked meaningfully different, and the whole thing was plausible enough that I went on to draw conclusions about an experiment I hadn't actually run as intended.
I'd qualify two parts slightly. The model that invented a wine was responding without the intended reference material, but whether inventing an answer was acceptable still depends on the instructions it received. And telemetry can be accurate about its observations without being sufficient to support the interpretation we place on them.
In fact, a later reader helped uncover another measurement problem in the throughput calculation, so even the instrumentation deserved more scrutiny.
But I completely agree with your central point: the empty reference block was visible in the results, and I read past it because the rest of the experiment seemed coherent.
That's probably the most reusable failure in the entire project. The system didn't need to hide the warning. It only needed to produce enough plausible surrounding evidence that I failed to recognize its significance.
Thanks for articulating that so well.
Oof, the invented wine got me. One model said NOT IN THE MATERIAL thirteen times while the other just made up a bottle, and every single call returned 200 through all of it. If you haven't already, worth logging the full prompt and asserting a sentinel string from the docs shows up in it, that catches this class of bug dead
Exactly. The HTTP 200s say nothing about whether the model received the material it needed to answer correctly.
I like the sentinel assertion as a regression test for that specific failure mode. It would catch the case where the expected source material never made it into the assembled prompt.
The qualification I'd add is that one sentinel doesn't prove the rest of the context arrived intact or that the answer was grounded in it. It's evidence that one boundary was crossed successfully, not proof of the entire chain.
And that's part of what made the invented bottle so revealing: the system could be operationally healthy while its evidentiary basis was missing. The model that repeatedly answered NOT IN THE MATERIAL was arguably behaving much more responsibly than the one that confidently invented an answer.
Your second failure mode has a version that never gets caught before the run, because it survives the entire history looking healthy, and the worst place for it to live is a score. A read-only scan today over the append-only event log we operate for AI agents pulled all 1,783 task-verdict rows. The numeric score field takes two values. 1.0 on 1,745 rows, 0.0 on 37. Those two sets are exactly the pass set and the fail set. So a float that reads like a graded quality measure is a re-encoding of the boolean sitting next to it, and ranking providers by mean score, which is the obvious thing to do with a float, only ranks them by pass rate.
The explanation fields complicate "absence does not raise" rather than just repeating it. Across those 1,783 rows the reasons field has five distinct values, and 1,744 of them are the same string. The evidence field, which exists to name event ids a reviewer could go re-check, is an empty array on 1,781 rows. All of those rows have the key. A presence check that tests for schema keys blesses every bit of this, and schema validation did exactly that for months, which is your kind five one layer in: verifying an artifact is present has to reach inside the container, since the key is not the content.
The thing I did not predict was the overlap. Exactly one row has non-empty evidence. Exactly one row carries itemized reasons instead of the constant. It is the same row. The decent behaviour did not spread thinly across fields, it collapsed into a single record, so the per-field numbers (99.9% empty, 99.9% constant) describe the shape badly. There is no gradient to walk up. There is one row that did the work and 1,782 that emitted a correctly shaped placeholder.
Your two new template rows are useful, but both get answered by reasoning about the harness before the run, and your own account gives that an uncomfortable record of zero for five. The data-side version is runnable after the experiment and before the write-up, without consulting what any field was meant to mean: for every column the write-up leans on, print its distribution, then measure how much of it another column in the same table already explains. Our score fails that in one line. Distribution alone would not have been enough, since two values heavily skewed toward 1.0 could plausibly mean the tasks were easy; the comparison against the pass bit is what exposes the duplication. It reaches your abstention collapse too, that being concentration in a single column, which is the check you ended up hand-building. The honest limit is that it inherits the weakness you name at the end. It tells you a column is degenerate and stays silent about why, so its output is a reason to go look upstream, never a diagnosis.
In the verdict table you already have, does any column still move once you hold the headline pass-or-fail result fixed?
The 360M model scoring 3 on confident wrong while escalating 43 of 50 is a trap I'd walk straight into, since the number moves the way everyone wants it to. The abstention check worries me more, because it fired for the right reason and still sent you to the wrong layer. Would it help if each warning printed what it could actually see, so the QA collapse message said it had only read outputs and never the prompt?
Yes, I think that would have made a substantial difference.
The abstention check detected a genuine pattern, but it had no visibility into the assembled prompt or retrieved reference material. It could tell me what the model returned, not why.
Having each warning declare its observation boundary would make that limitation explicit. Something like "13/15 abstentions observed; checker inspected responses only; prompt and retrieval contents not verified" would have been much harder to mistake for a diagnosis.
I particularly like that this doesn't require the checker to become more sophisticated at interpreting results. It requires it to be more precise about what it actually knows.
That's a useful design change for the harness, and it fits a larger principle I've been working through: evidence should carry its own visibility limits rather than leaving readers to infer them.
Thanks for the suggestion.
Had the same thing with a contact form on a shop I took over. Green success message, messages went nowhere, and nothing ever errored. How do you catch it when the wrong number looks normal?