Claude Haiku 5.5 came out on 7 October. Within a day I ran three popular Claude Code skills on it, and on Haiku 4.5 next to it. Three runs each, same task, same grader, same Claude Code version.
Here's the result that made me laugh.
The git workflow skill on Haiku 5.5, scored 0 to 1:
| Run | Without the skill | With the skill | Gap |
|---|---|---|---|
| 1 | 0.300 | 0.849 | +0.549 |
| 2 | 0.849 | 0.839 | -0.010 |
| 3 | 0.613 | 0.848 | +0.235 |
Run 1 is the screenshot that goes viral. "This one skill makes Haiku 5.5 55 points better."
Run 2 says the skill does nothing.
Same setup both times. And look at where the swing actually is: the skill arm sits at about 0.84 every time. It's Haiku 5.5 without the skill that jumped from 0.300 to 0.849 between runs. So the "+55 points" was mostly one bad baseline run.
Run it once and you would have published a fake win, with a real number attached.
The rest of it
- No skill clearly helped Haiku 5.5 in two or more of its three runs.
- The documentation skill clearly helped Haiku 4.5 in two of three runs (+0.14, +0.19, then +0.13 that was too noisy to call). On Haiku 5.5, all three runs had too few answers to tell either way. So does it still help on the new model? I can't tell you. Neither can anyone who ran it once.
- The code review skill on Haiku 4.5: run 1 said +0.34, clearly helped. Runs 2 and 3 said +0.09 and +0.02, too few answers to tell.
- Four of the six model and skill pairs changed their verdict between repeats.
That last line is the whole story. A lot of "I tried skill X and Claude got way better" posts are one run. Some popular skill-eval tools run a single attempt by default. Any one-run test would have reported run 1 above as a big win.
What this does not show
One task per skill, and only 3 to 10 answers per side in each run, so "too few answers to tell" means these runs couldn't see a 0.05 gap. It does not mean the skill is useless. The Haiku 5.5 calls also ran with Claude Code's per-turn effort setting and the Haiku 4.5 calls didn't, so the two models differ in more than the model. Every answer was scored three times by Claude Opus 5. This is not a model ranking.
The skills are from Addy Osmani's agent-skills pack, and I'm not knocking them. The point is that one run can't tell you whether any skill works, good or bad.
Full report, every number and every receipt: driftproofhq.com/reports/014
So, honest question: how many times do you run a skill before you decide it works?
*I maintain Driftproof, the open-source Claude Code plugin that ran this.
Top comments (5)
The table makes the point well. The skill arm sat near 0.84 in all three runs, and the whole swing came from the baseline (0.30, 0.85, 0.61). So the number that needs repeating most is the run without the skill.
If runs are limited, that suggests spending more of them on the baseline than on the skill arm, and reporting the baseline's spread next to the gap. A reader can then see whether "+0.55" is a strong skill or one weak baseline.
One question on the comparison: you note the Haiku 5.5 calls used the effort setting and the 4.5 calls did not. Did you try 5.5 at more than one effort level? It would help separate the model from the setting.
Agreed on both counts. The with-skill arm was the stable one in every run, so the baseline is the number that needed the most repeats, and this run gave both arms the same number of draws. Putting more of a fixed budget on the noisier arm is the right call, and it's on the list: a draw planner that tells you how many draws each arm needs before you spend. For now the receipt behind each cell shows both arms' spreads; the summary table on the report should show the baseline's spread next to the gap too, and I'll add that.
On effort: no, one setting only. This was a release-day run, so Haiku 5.5 got Claude Code's default per-turn effort for that model and Haiku 4.5 got none, because that's how Claude Code treats each. That's exactly why the report says the two Haiku arms differ in more than the model. Running 5.5 at a second effort level is the right next experiment, and it's a cheap one; if I do it, it goes in as an amendment to the same report rather than a new headline.
The swing being entirely in the baseline arm is the pattern that breaks most agent evals. The skill held the model steady at roughly 0.85 across all three attempts, providing an execution rail that kept runs from spiraling on dirty git worktree states.
When a small model fails an unassisted agent loop, the failure is rarely an incremental score dip. An early bad command sends it down a catastrophic branch where it burns its remaining turns trying to recover. A skill bounds that floor. Evaluating against a single baseline run measures the unassisted model's tail risk instead of any capability increase.
You're half right, and the half you're right about is the dramatic one. In the git-workflow pair the swing is all baseline: without the skill the three runs scored 0.300, 0.849 and 0.613 (spread 0.549), with the skill 0.849, 0.839, 0.848 (spread 0.010). That matches your guardrail reading exactly: one early bad command and the bare model goes down a failing branch, and the skill mostly removes that branch. The +55/nothing headline is runs 1 and 2 of that pair.
But code-review flips it: baseline spread 0.010, with-skill 0.121. There the skill is the noisy arm, so "the swing sits in the baseline" is a per-skill fact, not a law. We ran into your point before, report 007 is literally titled "the baseline is the noisy arm": driftproofhq.com/reports/007/
Fully agree on the conclusion though. A single baseline run is a sample from the tail, not a measurement, which is why our verdicts only ever come from ranges over repeats, never one pair of numbers.
Non-determinism at that scale usually comes down to three things: temperature sampling variance (even at 0 there's floating-point drift across runs), context window truncation differences when the skill pulls in repo files, and the model's internal "reasoning" path branching on ambiguous instructions. Haiku 5.5's shorter context vs Sonnet makes it more sensitive to which files get stuffed into the prompt each run.
What helped us stabilize similar evals: pin the exact file list the skill receives (hash the file tree, fail if it changes), force a deterministic seed via the API if the provider supports it, and add a "self-consistency" pass — run the skill 3×, take majority vote on the diff. Costs more tokens but turns "sometimes works" into "works or clearly fails."
Also worth checking if the skill uses any tool calls with side effects (file writes, shell commands) — those can leave repo state that pollutes run 2. We wrap each eval in a fresh git worktree now (site: labagent .tech)