I wanted to know whether a linter for AI writing tells helps an agent rewrite a file better than a good prompt does. So I ran a benchmark with 11 a...
For further actions, you may consider blocking this person and/or reporting abuse
The held-out result is the honest part of this post, and it is why I trust the method more than the headline — on the public set your linter wins by 2.67, but on the held-out set four arms sit within a point of each other, which tells me the tasks stopped separating the tools rather than the tools getting worse. I have seen the same shape in my own writing: a piece that scores well on the examples I trained it against and falls apart on a fresh topic, because the test was measuring familiarity, not quality. What would a held-out set need to look like to actually separate these tools — harder files, or files from writers the tools have never seen?
Thank you, that is the right reading. The held-out set hit the ceiling: four arms scored 29.67 or more out of 30, so it could not separate anything. It says the tasks were too easy, not that any tool got worse.
The public set has the problem you describe. I developed against it, so part of that 2.67 can be familiarity.
From what went wrong here, I think a set that separates these tools needs three things:
I tried the first two on another tool of mine, surviving-lines. A writer who never saw the repository wrote the sealed set, and every arm scored low; the best scored 9.67 of 25. That set did separate the tools, but only just: the interval of the gap was +0.33 to +4.33. So harder files help, and they also show how small the real gap can be.
If you have a piece that fell apart on a fresh topic, I would like to try it as a task.
The sealed-set result is the most useful number here: every arm dropping to 9.67 out of 25 tells me the public set was not just familiar, it was too easy to separate anything. But I would add a fourth requirement to your three, which is a pre-registered rubric. If the sealed writer also writes the scoring criteria before any tool runs, you avoid the quiet leak where a rubric is softened after seeing what the tools actually produced. The interval you report, +0.33 to +4.33, is also a warning: even with harder files, six tasks and three runs each leave a gap that could be mostly noise. Would you rather add more tasks first, or more runs per task, when the cost of a run is the binding constraint?
Agreed, a pre-registered rubric belongs on that list. In the later rounds the writer who never saw the code wrote the criteria together with the tasks, both were sealed, and the protocol was frozen before the first cell. I still broke my own rule once: I changed the grading from three votes to five in the middle of a round, and one held-out interval moved from [0.33, 3.33] to [0.00, 3.33]. I published both versions, but that is exactly the kind of leak you describe.
On your question: more tasks first. Extra runs only shrink the noise inside a task, and here most of the spread came from the differences between tasks. With six tasks, each task has a large effect on the interval, so a seventh and eighth task buy more than a fourth run.
Publishing both versions is the right remedy for the mid-round grading change — the leak is disclosed rather than hidden, and the reader can see the interval move from [0.33, 3.33] to [0.00, 3.33]. The trap I would watch with the seventh and eighth tasks is selection: new tasks chosen because they separate the current tools reintroduce the familiarity leak in miniature, just at the task level instead of the run level. Do the next tasks come from writers who have never seen any of the tools, or from the same sealed pool that wrote the first six?
Good catch. If I pick tasks because they separate the tools I already have, I am tuning the test to the result, only one level up.
So the next tasks will come from a new writer who has seen none of the tools, none of the code and none of the results so far. That writer also writes the criteria, and both are sealed before any run. Every task they produce goes in, including tasks where all the tools tie. A tie is a result too, and dropping it would bring back the selection you describe.
The cost is that a new set may separate nothing again. I would rather publish that than a set I shaped.
Twelve tasks is a small sample for a gap of 2.67. The number I would add is the tokens each run spent next to the score: a rewrite pass adds turns, so a better file bought with extra turns is a different result.
You are right that twelve tasks is a small sample. That is also why the held-out result matters: on the second set the top tools tied, so I do not read the 2.67 as a general win.
The tokens point is fair too. A rewrite pass adds turns, so a better file can just be a more expensive one. The run logs record the cost of every call, so the cost per arm can sit next to the score. It is not in the article yet, and I will add it.
Thanks for the article, it prompted me to update my own writing rules, in particular checking that a rewrite preserves meaning. Have you measured how often --compare catches changes that the agent’s own review misses?
the held out result is the part i trust more. four arms at 29.67 or higher with the no plugin baseline at 28.67 means those tasks cannot separate anything, and you said so instead of claiming first place.
we hit the same ceiling effect with an eval set that was too easy. baseline sat at 94% and every prompt change looked flat until we added tasks where facts break under a rewrite. the compare step that lists lost numbers and qualifiers is the useful bit imo.
how often did it flag a rewrite that scored full marks on the scripted checks?
@efe_genc Yes, cost per task next to the score, not cost per run. A rewrite pass that solves more tasks for the same spend is a win; one that only spends more is not.
Agreed, cost per task is the better unit. A rewrite pass that solves more tasks for the same spend is a gain, and one that only spends more is not. I will put cost per task next to each arm's score. One limit I will state with it: the cost is computed at list price from the logged usage, because the runs were charged to a subscription, not billed per call.