Before I Compared the Models, I Had to Decide What Not to Count

Originally published on hexisteme notes.

I run several models across a fleet of agent CLIs, and I wanted a plain question answered: on my own work, which ones re-edit the same file more, which ones burn more validation runs, which ones error on tool calls more often. I already log every turn, so the measurement looked like a GROUP BY.

It wasn't. The query was an afternoon. The honesty was the week. Almost everything I did after the first GROUP BY was a decision about what not to count, because every shortcut I could take would have produced a confident number that was wrong. This is that list.

The average of two roles is not a number about either one

Each turn in my logs belongs to a role: a main thread, where a model works interactively on a long task, or a sidechain, a short one-shot the harness spins off. These are not two samples of the same thing. A main thread for one model had a median re-edit rate near 0.44; its sidechains sat at 0. Pool them and the model's "re-edit rate" becomes an artifact of how many sidechains happened to run that week — a number that moves when the workload mix moves, not when the model changes.

So the unit of comparison is model × role, never model alone. Every table is stratified by role, and I do not compare a main cell to a sidechain cell even for the same model: the session length and structure differ too much to read the gap as anything but confounding.

Different regimes are different treatments, so I don't pool them

The same model name covers more than one thing over time. One of mine ran under a pre-launch configuration, then a relaunch, with a blocked window between. Those are not one population with more data; they are distinct treatments. Averaging across them hides exactly the shift you built the lab to see. I keep them as separate rows — model/pre and model/relaunch never get summed — with epoch boundaries pinned to calendar dates so the split is reproducible and not drawn to flatter a result.

A floor for the word "comparable"

Two thresholds, declared before I looked at the outcomes:

• n ≥ 5 to appear as a comparable cell. Below that I print NOT_COMPARABLE, not the value. One cell had exactly one observation. Its median is a real number and a useless one; setting it beside a cell with 2,781 observations invites a comparison the sample cannot support.
• n ≥ 20 on both sides before I'll compute a confidence interval for a difference.

The point of writing the floors down first is that otherwise you discover them afterward, at whichever value makes the cell you like look solid.

The rows I refused to count

871 of my threads — 57 main, 814 sidechain — could not be attributed to any model. They were synthetic harness activity with zero tool and token behavior. The tempting move is to treat "unknown" as "zero" and fold them in. That single substitution would have shifted every rate and every denominator, because missing is not the same as zero and an unattributed turn is not a well-behaved one.

They appear in exactly one place: a reference line that says excluded, 871, here is where they went. They enter no average. A measurement that silently drops what it can't classify is the one that lies first, so I made the drop loud and countable instead.

When a confidence interval of [0, 0] is not confidence

Most of these per-turn metrics are zero most of the time. A model does not re-edit on most turns; it does not error on most turns. When I bootstrapped a median difference between two such cells, the interval frequently came back [0, 0].

That looks like a precise result. It is the opposite. When the data is almost all ties at zero, the median of every resample is zero, so the interval collapses to a point — not because the estimate is sharp but because the statistic has nothing to vary over. I label those collapses as a coverage limitation of a median bootstrap under heavy ties, not as a tight finding. A degenerate interval and a precise one print the same two characters; only the sample behind them tells you which you have.

The tokens that look additive but aren't

I wanted a total-output comparison, so I summed output tokens per turn. That one is genuinely additive. The trap was next to it: the stored input-token and cache-read counts are the context re-sent on every turn, counted again each time — a running sum with the same tokens in it over and over, not a count of distinct tokens. Summing those to compare "how much each model read" would have rewarded whichever model took more turns. They are logged; they are not a total. Only the output sum answers the question I actually asked.

The metric is not the thing it is named after

Two labels in my own schema promised more than they measure, and I wrote the gap down next to each:

• same_file_reedit_rate counts a file touched again. It does not separate iterating toward a result from reworking a mistake. The name suggests a quality signal; the measurement is a frequency. If I ever make it a primary metric, that limitation gets pre-registered with it.
• completion_proxy reads the shape of a thread's last line. It is a heuristic for "finished," not a judgment that the work was done, and I had to censor threads whose snapshot was cut off mid-turn so a truncation wouldn't get scored as an abandonment.

The measurer is inside the measurement

The sessions in which I built and audited this pipeline are in the corpus it measures. Those turns land in one project's stratum and distort its numbers — the tool is watching itself work. I can't retroactively remove the observer from the observed, so the limitation is named in the report and the next version adds a stratum that excludes self-measurement. Naming it is not fixing it; it's refusing to let the contamination pass as a finding.

Association, not cause

Models were never randomly assigned to tasks. Harder work, different projects, different weeks, changing dispatch policy — all of it is confounded with which model ran. So the whole thing is graded association-only and exploratory. There are no p-values; the comparison answers what co-occurred, not what caused what, and any sentence that drifts toward "model X is better" is one the design cannot support.

What this does not prove

• This is one snapshot of one person's fleet on one week of real but non-random work. It is not a benchmark, and it does not rank models on quality.
• The role split removes one confounder (session structure) and leaves the rest — task difficulty, project, timing, and policy changes are all still tangled with model assignment.
• The sequence metrics are computed inside a model's own epoch, so work that crosses a model switch — one model's edit validated by another — is not charged to either.
• "Thread" and "session" are not the same count; one session spans several files, and the attribution unit here is the per-turn epoch, not the session.
• Every threshold (n ≥ 5, n ≥ 20, the purity cutoff for proxy threads) is a choice. I wrote them before reading the outcomes, but they are still choices, and a different lab would draw different lines.

The deliverable here was never the ranking table. It was the list above — the specific numbers I declined to print, and the reason each one would have been a lie. A comparison is mostly the rows you refuse to count.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.