I wanted to know whether a linter for AI writing tells helps an agent rewrite a file better than a good prompt does. So I ran a benchmark with 11 arms: the linter packaged as a Claude Code plugin, six other Claude Code skills that claim to remove AI-sounding prose, and Claude with no plugin. Three of those skills (Humanizer, unslop and everwrite) were also run a second time with the prompt calling them by name, which is where the count of 11 comes from. Each arm ran each task three times on Sonnet 5.5.
The short result: on 12 public tasks the linter scored 58.67 of 60 and the best other arm scored 56.00. On 6 held-out tasks, written by an author who had not read the code, four arms scored 29.67 or more of 30, so that set does not separate them.
What was measured
Each task asks an agent to edit a real file in place: a French launch note, a README intro, a pull request description, release notes, a mixed-language guide, a repository set-up. A task scores only what the file contains afterwards, so talking about the problem earns nothing. Each task also carries facts a careless rewrite would break, such as numbers, names, code and links, and scripted checks look for them. Some criteria need judgment, and those went to a blind grader: outputs of all arms were shuffled and anonymised, and the grader gave its reason before its verdict, five votes per cell. Of 99 blind cells, 92 were unanimous.
The other six tools are Humanizer (blader/humanizer, about 60,000 GitHub stars per the GitHub API on 2026-10-24), unslop, everwrite, slopless, avoid-ai-writing and slop-cop. The protocol was written and committed before the first cell ran. A total below is the sum over tasks of the mean over the three runs, with a maximum of 5 points per task. The result applies to those tasks only and ranks nothing in general.
Public tasks
| arm | total of 60 | 95% interval |
|---|---|---|
| this linter | 58.67 | [57.67, 59.67] |
| Humanizer, called by name | 56.00 | [54.67, 57.33] |
| Humanizer | 54.67 | [53.33, 56.33] |
| everwrite, called by name | 54.33 | [52.67, 56.33] |
| everwrite | 53.33 | [52.00, 54.67] |
| unslop, called by name | 51.67 | [51.00, 53.00] |
| unslop | 51.00 | [50.33, 51.67] |
| slopless | 50.67 | [49.33, 52.00] |
| avoid-ai-writing | 50.33 | [49.33, 51.67] |
| slop-cop | 49.00 | [47.67, 50.33] |
| no plugin | 48.67 | [47.00, 50.00] |
The difference between the linter and the best other arm, Humanizer called by name, is 2.67 points, with a 95% bootstrap interval of [1.00, 4.33]. The lower bound is above zero.
It did not win every task. On h11 and h15 it scored 4.67 where the best rival scored 5.00, and on h18 it tied at 4.67. The benchmark's stricter per-task rule (rule B) fails on those three tasks.
Held-out tasks
Six further tasks were written by an author, a Sonnet agent, who had seen no code of this repository. The person running the measurement had not read them before it.
| arm | total of 30 | 95% interval |
|---|---|---|
| unslop | 30.00 | [30.00, 30.00] |
| avoid-ai-writing | 29.67 | [29.00, 30.00] |
| this linter | 29.67 | [29.00, 30.00] |
| unslop, called by name | 29.67 | [29.00, 30.00] |
| no plugin | 28.67 | [28.00, 29.00] |
| Humanizer | 28.67 | [28.00, 29.00] |
| Humanizer, called by name | 28.00 | [27.00, 29.00] |
| slopless | 28.00 | [28.00, 28.00] |
| everwrite, called by name | 27.33 | [26.33, 28.33] |
| slop-cop | 27.00 | [25.67, 28.33] |
| everwrite | 26.67 | [26.00, 27.00] |
Here the linter is 0.33 behind unslop, with an interval of [-1.00, 0.00], so the claim that it is first does not hold on this set. It lost that 0.33 on one task, ho1, where it scored 4.67 against 5.00. Four arms sit at 29.67 or higher and even the no-plugin baseline reaches 28.67, so these tasks cannot separate the top arms. Across both sets: first on the public set, and within 0.33 of the best of the 11 arms on the held-out set.
Other models
I also ran Haiku 5.5 and Opus 5.5 once per task, with only the linter, the best other arm of each set and the no-plugin baseline. These totals are descriptive. With one run per task, a difference of one or two points is inside the run-to-run variation. These runs were not blind-graded, so the public scores use the scripted criteria only and their maximum is 57.
| model | set | this linter | best other arm | no plugin |
|---|---|---|---|---|
| Haiku 5.5 | public, of 57 | 54 | 52 (Humanizer, called by name) | 43 |
| Haiku 5.5 | held-out, of 30 | 28 | 28 (unslop) | 26 |
| Opus 5.5 | public, of 57 | 54 | 54 (Humanizer, called by name) | 45 |
| Opus 5.5 | held-out, of 30 | 29 | 27 (unslop) | 25 |
What a prompt cannot do
The linter is a command line tool that calls no model and has no dependencies. Its rules are deterministic: the same text gives the same findings. A rewrite can pass every check and still lose a fact, so the tool also has --compare. It lists each number, version, date, link, quotation, code span and name that appears in one text and not the other, and exits 1 when it finds one. It also reports a qualifier that went missing next to a number, and first-person or claim words that the rewrite added.
This is real output on a two-line example I wrote for this post:
$ npx ai-slop-linter --compare before.md after.md
facts: 1 kept, 4 lost, 0 added
lost date 3 March 2026
lost number 1200
lost qualifier up to 40
lost version 2.4
The plugin's skill runs this after each rewrite and treats a failing comparison as blocking. --compare compares facts and not wording, so it cannot say whether a rewrite is good.
What it costs
This arm cost 2.43 times the no-plugin baseline per cell at list price, on both sets, because it takes more turns and more tokens. In the public set that was 0.125 US dollars per cell against 0.051. The runs were charged to a subscription, and the whole measurement would cost about 91 US dollars at list price.
What went wrong in the measurement
- The session limit stopped the runs twice. 236 public and 160 held-out attempts failed with
usage_limit_reached, were kept and never scored, and the same commands ran them again. - The public and held-out runs went on at the same time. The held-out runs used a later commit, because the harness refuses a working tree whose benchmark folder differs from the commit. The product files are byte-identical between the two commits.
- The blind grader loop died once and was started again, so for a while two copies wrote vote files. All 165 vote files of the re-grade are valid, and none was dropped by hand.
- A re-grade of 33 sampled blind runs agreed with the first grading on 99.4 percent of criteria. Of 60 sampled cells run a second time, 38 of 40 public and 19 of 20 held-out re-runs fell inside the range of the original three runs.
Try it and check it
npx ai-slop-linter README.md
npx ai-slop-linter --compare before.md after.md
To install it as a Claude Code plugin:
/plugin marketplace add Bubblegunn/ai-slop-linter
/plugin install ai-slop-linter@ai-slop-linter
Every cell, the departures from the protocol and a recompute step are in the round 8 evidence folder. REPRODUCE.md in that folder gives the commands for a re-run, which needs Claude Code 2.1.308, Node 24 and about 9 hours. The tasks, criteria and harness are public in bench/head-to-head/, and the earlier rounds and rulings are in docs/HEAD-TO-HEAD.md. The repository is Bubblegunn/ai-slop-linter, and version 0.2.0 is on npm.
Two people outside my own commits have had pull requests merged into the linter. Aaqibhafeez Khan (@Aaqibhafeezkhan) wrote the SARIF output for code scanning, the pre-commit hooks and the rule for leftover reference markup. Shivam Gupta (@shivam-070208) tested Hindi against a corpus and wrote down that it produced no rule safe to ship, which is why the linter makes no claim about Hindi. Thank you both.
Disclosure: I drafted this article with AI assistance (Claude) and reviewed it myself. Every number comes from the evidence folder linked above, and the benchmark can be re-run from it.
Top comments (19)
The held-out result is the honest part of this post, and it is why I trust the method more than the headline — on the public set your linter wins by 2.67, but on the held-out set four arms sit within a point of each other, which tells me the tasks stopped separating the tools rather than the tools getting worse. I have seen the same shape in my own writing: a piece that scores well on the examples I trained it against and falls apart on a fresh topic, because the test was measuring familiarity, not quality. What would a held-out set need to look like to actually separate these tools — harder files, or files from writers the tools have never seen?
Thank you, that is the right reading. The held-out set hit the ceiling: four arms scored 29.67 or more out of 30, so it could not separate anything. It says the tasks were too easy, not that any tool got worse.
The public set has the problem you describe. I developed against it, so part of that 2.67 can be familiarity.
From what went wrong here, I think a set that separates these tools needs three things:
I tried the first two on another tool of mine, surviving-lines. A writer who never saw the repository wrote the sealed set, and every arm scored low; the best scored 9.67 of 25. That set did separate the tools, but only just: the interval of the gap was +0.33 to +4.33. So harder files help, and they also show how small the real gap can be.
If you have a piece that fell apart on a fresh topic, I would like to try it as a task.
The sealed-set result is the most useful number here: every arm dropping to 9.67 out of 25 tells me the public set was not just familiar, it was too easy to separate anything. But I would add a fourth requirement to your three, which is a pre-registered rubric. If the sealed writer also writes the scoring criteria before any tool runs, you avoid the quiet leak where a rubric is softened after seeing what the tools actually produced. The interval you report, +0.33 to +4.33, is also a warning: even with harder files, six tasks and three runs each leave a gap that could be mostly noise. Would you rather add more tasks first, or more runs per task, when the cost of a run is the binding constraint?
Agreed, a pre-registered rubric belongs on that list. In the later rounds the writer who never saw the code wrote the criteria together with the tasks, both were sealed, and the protocol was frozen before the first cell. I still broke my own rule once: I changed the grading from three votes to five in the middle of a round, and one held-out interval moved from [0.33, 3.33] to [0.00, 3.33]. I published both versions, but that is exactly the kind of leak you describe.
On your question: more tasks first. Extra runs only shrink the noise inside a task, and here most of the spread came from the differences between tasks. With six tasks, each task has a large effect on the interval, so a seventh and eighth task buy more than a fourth run.
Publishing both versions is the right remedy for the mid-round grading change — the leak is disclosed rather than hidden, and the reader can see the interval move from [0.33, 3.33] to [0.00, 3.33]. The trap I would watch with the seventh and eighth tasks is selection: new tasks chosen because they separate the current tools reintroduce the familiarity leak in miniature, just at the task level instead of the run level. Do the next tasks come from writers who have never seen any of the tools, or from the same sealed pool that wrote the first six?
Good catch. If I pick tasks because they separate the tools I already have, I am tuning the test to the result, only one level up.
So the next tasks will come from a new writer who has seen none of the tools, none of the code and none of the results so far. That writer also writes the criteria, and both are sealed before any run. Every task they produce goes in, including tasks where all the tools tie. A tie is a result too, and dropping it would bring back the selection you describe.
The cost is that a new set may separate nothing again. I would rather publish that than a set I shaped.
That sealed-writer rule is the part I would hold you to, because sealing the criteria before any run is what stops the rubric from quietly moving toward the tool you already like. Publishing a set that separates nothing is still a result — it tells you the tasks were too easy or too narrow, not that the tools are equal. Will you keep the ties in the published set, even when every arm lands at the top?
Yes. Ties stay in the published set, including ties at the ceiling. The held-out set in this article is already one: four arms scored 29.67 or more out of 30, and it is published as it came out. A ceiling tie gets reported as what it is, a set too easy to separate the tools, and the next set is written to have more headroom. It does not get dropped or quietly replaced.
Publishing the ceiling tie as-is is the honest move, and it is also a diagnostic: four arms at 29.67 or more out of 30 tells me the rubric, not just the set, may be too coarse to separate them. For the next set, I would keep the total but also publish per-criterion misses, so a tie on the total can still show which tool missed the one hard fact in a long file and which missed nothing at all. That way more headroom is not only about harder tasks, it is about a finer ruler. Will the next set keep the same sealed-writer rule, with the writer never seeing any of the tools or code?
Yes, the same rule: a new writer who has seen none of the tools, the code or the results, criteria written with the tasks, both sealed before any run.
Per-criterion misses are a good addition. The graders already vote on each criterion separately, so the data exists; I only published the totals. For the next set I will publish a table of which criterion each arm missed, so a tie on the total still shows who dropped the one hard fact and who dropped nothing.
That combination settles it for me: a new writer sealed before any run, plus a table showing exactly which criterion each arm missed, turns a 29.67 tie from a shrug into evidence. In my own reviews I have seen two tools tie on a total while one missed a single factual check in a long document and the other missed nothing, and only the per-criterion view made that difference visible. Will the table list the missed criterion by name for each arm, rather than just a count of misses?
Yes, by name. Each row will be an arm, each column a criterion as the writer worded it, and each cell will show how many of the five votes marked it missed. A count of misses alone would hide exactly the difference you describe: two arms with one miss each, where one dropped a formatting detail and the other dropped the one hard fact.
the by-name breakdown is what makes the table work. a bare miss count would hide the formatting-versus-fact split.
Twelve tasks is a small sample for a gap of 2.67. The number I would add is the tokens each run spent next to the score: a rewrite pass adds turns, so a better file bought with extra turns is a different result.
You are right that twelve tasks is a small sample. That is also why the held-out result matters: on the second set the top tools tied, so I do not read the 2.67 as a general win.
The tokens point is fair too. A rewrite pass adds turns, so a better file can just be a more expensive one. The run logs record the cost of every call, so the cost per arm can sit next to the score. It is not in the article yet, and I will add it.
Thanks for the article, it prompted me to update my own writing rules, in particular checking that a rewrite preserves meaning. Have you measured how often --compare catches changes that the agent’s own review misses?
the held out result is the part i trust more. four arms at 29.67 or higher with the no plugin baseline at 28.67 means those tasks cannot separate anything, and you said so instead of claiming first place.
we hit the same ceiling effect with an eval set that was too easy. baseline sat at 94% and every prompt change looked flat until we added tasks where facts break under a rewrite. the compare step that lists lost numbers and qualifiers is the useful bit imo.
how often did it flag a rewrite that scored full marks on the scripted checks?
@efe_genc Yes, cost per task next to the score, not cost per run. A rewrite pass that solves more tasks for the same spend is a win; one that only spends more is not.
Agreed, cost per task is the better unit. A rewrite pass that solves more tasks for the same spend is a gain, and one that only spends more is not. I will put cost per task next to each arm's score. One limit I will state with it: the cost is computed at list price from the logged usage, because the runs were charged to a subscription, not billed per call.