I ran a small pilot before the real experiment. It turned up more than ten bugs.
Every one of them was in my own measurement code. Not one of them taught me anything about the question I was actually trying to answer.
This is a follow-on to killcheck, the test-quality harness I wrote about here earlier this year. The new work is a research paper entered into a Kaggle competition about AI coding agents built on Google's Gemma 4 models, looking at how you tell whether an agent's bug fix is actually correct. It submits in November and the results come out after the competition closes, so this post has no findings in it. It has the part that came before the findings, which I think is the more useful half anyway.
The bugs
They were boring. That is what makes them dangerous.
Ordering that was not stable between runs, so the same input produced a different sequence depending on how the filesystem felt that morning. Randomness I had not seeded and did not know was there. Timeouts that fired under load and were recorded as a result rather than as a missing result. Output that simply differed between two identical runs for reasons I could not immediately explain.
None of those announce themselves. None of them crash. They all produce a number, and the number looks exactly like a real number, and you can put it in a table and nobody will blink.
I had written about this pattern after the last project. Then I went and did it again on the new one. Knowing the failure mode exists turns out to be almost no protection at all.
The one trick that worked
Here is the thing that found most of them.
Build controls whose answer you already know.
The clearest example: a change that does nothing should always score zero. If you feed your pipeline something inert and the pipeline reports a result, the result came from your pipeline. There is nothing else it could have come from. You have just caught your instrument manufacturing a number out of nothing.
Mine did not score zero. More than once.
That is the whole technique and it is almost embarrassingly simple. You are not testing the thing you are curious about. You are testing whether the machine that measures it can correctly measure something you already know the answer to.
If it cannot, every number it has ever given you is unverified. Not wrong necessarily. Unverified, which in practice means the same thing when you are about to publish.
Why this is different from just testing your code
I had tests. The tests passed.
Tests check that your code does what you told it to do. A control checks whether the thing it produces corresponds to reality. Those are different questions, and on measurement code the second one is the one that matters, because the failure mode is not an exception. The failure mode is a plausible number.
A plausible wrong number has no symptom. It does not throw. It does not look odd in a log. It sits in your results table being quietly incorrect for as long as you let it, and the only thing that ever catches it is a case where you knew the answer in advance and can see that the machine disagreed with you.
This also explains why the bugs clustered where they did. Everything in my pipeline that had an obvious right answer got checked early, because checking it was easy and the answer was unambiguous. Everything that produced a judgement call went unchecked for much longer, because there was nothing to check it against. So I built something to check it against.
The order this forces on you
What I would do differently, and what I did do this time once I had learned it the hard way, is to build the controls before the experiment rather than after the first confusing result.
It feels like a delay. You want to get to the interesting part. The controls are not the interesting part, they produce no insight, and nobody reads a paper for the section where you confirm your tooling can count to zero.
But every hour spent on them up front is an hour you do not spend later trying to work out whether a surprising result is a discovery or a defect. That second question is much more expensive, because by then you have an opinion, and opinions make you lenient with evidence that agrees with you.
The lesson
Before you trust a number about someone else's system, make your instrument prove it can produce a number it already knows.
A no-op scores zero. An empty input gives an empty result. An identical rerun gives an identical answer. These are not deep insights. They are the floor. But the floor is where the whole table stands, and I have now twice found out the hard way what happens when you skip it.
More than ten bugs before I learned a single thing about the actual research question. I think that ratio is normal. I think most people just do not count.
Top comments (4)
The inert-input control is the right first move. There's a class it can't reach: the instrument can be correct at the granularity you ran the control and still produce a wrong published number, because the combining step sits between the two.
Concrete instance from my own work, since the failure is more useful than the principle. I had a table of results over 56 seed cells and published the sentence "the gap column is exactly half the tie share in every cell." That was a property of the per-cell quantity, and it was exactly true: 56/56 cells, worst deviation 7.5e-15, algebraic rather than fitted. Everything upstream was seeded and deterministic, reruns byte-identical — a control at that level passes on every run.
The four numbers printed beside the sentence were still wrong. Each column had been aggregated independently (median of that column across the cells), and
median(half) − median(strict) ≠ median(half − strict). My inert-input control ran on the per-cell values and returned zero by construction; the ratio column I published was computed from the aggregated columns and read 0.40 / 0.52 / 0.50 / 0.58 instead of the exact 0.5 the sentence claimed.It surfaced when the ratio was computed independently in a reply: the numbers agreed with each other and disagreed with the sentence. That agreement is worth naming, because it isn't a check — two computations that share the aggregation step will confirm each other's arithmetic and share its error.
How easy that is to miss, measured: with an identity that holds to 3.6e-15 in every cell, independently aggregated columns reproduce the exact ratio in 2 of 200 random tables; the rest land anywhere between 0.15 and 0.76. The per-cell identity isn't a weak signal — it's a different object from the table.
The fix was two lines: recompute the derived column from the same cells instead of from the aggregated columns. The repaired row is exactly 0.5× the tie column.
So the addendum I'd put under your control: an inert input exercises the path up to the number the pipeline returns. If what you publish is a combination over runs, seeds, agents or cases — a median, a mean, a rate, an improvement percentage — the control has to be run on that combination, not on the runs. A zero per run licenses nothing about the aggregate. It doesn't shrink with more runs either, which makes it a different axis from the noise floor discussed above.
Cheap version: take an identity your per-cell quantity satisfies by construction, verify it in the cells and in the table, and treat a mismatch as a bug in the table, not in the identity. And the uncomfortable half of your opening: "output that differed between two identical runs" is only the visible case. A number can be deterministic, seed-stable, reproducible from your own script, and still not measure what the sentence around it says.
This is the gap, and the example is what makes it land rather than the principle.
What my control actually certifies is narrower than I wrote it. An inert input proves the path from input to per-unit number is not fabricating. It says nothing about the step after that, and the step after that is the one that produces the sentence.
The generalisation I would pull out of your case: the problem is non-linear aggregation specifically. Means and sums commute with differences, so a per-unit control carries through to them. Medians do not. Ratios of aggregates do not. Rates with a moving denominator do not. So the rule is sharper than “controls must run on the aggregate”: if the published number is a non-linear function of the per-unit numbers, there is no per-unit control that reaches it, and no amount of care upstream helps.
The line I want to take away and keep is this one: two computations that share the aggregation step will confirm each other’s arithmetic and share its error. That is the failure mode sitting under most of what gets called independent verification. Recomputing a number a second way, in a second language, by a second person catches nothing if both paths run through the same wrong step. Independence has to be at the step that is wrong, and you do not know which step that is in advance. Which is why the identity check is the right move rather than a second derivation: it does not recompute the number, it tests a property the number is required to have.
The 2 of 200 figure is the part I would have got wrong on intuition. I would have read an identity holding to 1e-15 in all 56 cells as strong evidence about the table. It is not weak evidence about the table. It is evidence about a different object.
Taking the cheap version, and adding one step to it for my own use: list every number that will appear in the published text and mark it as per-unit or combination before writing the sentence around it. The second column is the set with no control on it. I have work in flight where this applies and I am going back through it before anything is final. If something moves, I will say so here.
And the closing line is the one that stings. Deterministic, seed-stable, reproducible from my own script. All four are properties of the computation. None of them is a property of the claim.
The part that made the no-op control useful for me is that it also measures how much of any later difference is noise. A zero-score control run N times with zero hits only bounds the false-positive rate: 20 clean runs leave an upper limit of about 14% (95%, exact), 50 runs about 6%, so "it scored zero once" says little about an unstable pipeline.
Two cheap additions to the list at the end: run the identical-rerun control with the seeds and ordering you did not pin, and keep the spread of those reruns as the noise floor. Then a comparison between two agents or two Gemma sizes counts only if the gap is bigger than that spread, not just nonzero.
On the timeouts you recorded as results: a third control, where you force a timeout on a case with a known answer, would show whether the harness reports "missing" or a score. Did the more than ten bugs fall mostly in one stage, scoring or execution?
The rule of three is the correction I needed. Zero hits in N runs bounds the false-positive rate at roughly 3/N, and I ran mine nowhere near enough times to claim much. “It scored zero” was doing more work in my post than it earns. A control that has passed once is a control you have sampled once.
The noise floor point is the one I am taking straight into current work. Rerun with the seeds and ordering I did not pin, keep the spread, require any comparison to clear it. What that changes for me conceptually is that I had been treating stability as something to achieve and then stop thinking about, rather than something to measure once and then carry forward as a threshold. A nonzero gap becomes a candidate rather than a finding.
The forced-timeout control is a third kind and it is worth naming why, because it took me a moment to see it. The inert input tests the scoring path. The rerun tests stability. The forced timeout tests the recording path: whether a case that produced no answer gets recorded as no answer or as an answer of zero. That is neither a scoring bug nor a stability bug. It is a classification bug, and it contaminates the score without ever touching the scorer.
On your question, the honest answer rather than a tidy one: almost all of them were in execution and recording. Essentially none were in scoring.
I think that is expected rather than lucky. The scoring code is close to pure. Same inputs, same number, easy to test, and I tested it. The execution and recording code touches the filesystem, the clock, process scheduling and the outside world, and every one of my bugs lived on one of those boundaries.
Which leaves something slightly uncomfortable. The stage I was most confident about deserved that confidence. The stage I had filed as plumbing is where all of the damage was. I had been checking the interesting code.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.