Update, 22 Oct. After the comment from Skillselion below, the four router skills (find-skills, agent-browser, hyperframes, hyperframes-registry) are being taken out of the verdict counts and marked "not measurable with text tasks": a text task scores the router's written reply, not the skill it hands off to. That removes the original headline about find-skills, so the title has changed. The report amendment is being applied; until it is live, the report page and the scoreboard below still show the original counts for all 20. The receipts are unchanged.
skills.sh ranks agent skills by installs. Installs tell you a skill is popular. They don't tell you whether it changes what the agent actually writes.
So we took the top 20 skills from the skills.sh snapshot of 17 Oct and ran each one twice on the same tasks: once with the skill loaded, once without. Two answering models, Claude Sonnet 5.5 and Claude Opus 5.5, with Claude Haiku 5.5 as the judge. 40 results in all.
The scoreboard
| Sonnet 5.5 | Opus 5.5 | |
|---|---|---|
| Clearly helped | 12 | 12 |
| Clearly hurt | 4 | 2 |
| Too few answers to tell | 4 | 6 |
11 of the 20 skills clearly helped on both models.
What hurt
- find-skills (vercel-labs/skills), number one by installs: clearly hurt on both models. Both times it was the same task, a search request, where answers scored about 0.75 with the skill against 0.92 to 0.95 without. find-skills is a router that points the agent at other skills. We tested its written answer, not the skills it hands off to.
- reddit-automation (flowkit-labs/skills): clearly hurt on both models, on the ranking task, and on the reply task with Sonnet.
- grill-with-docs and hyperframes: clearly hurt on Sonnet only.
What helped most
- hyperframes-registry (heygen-com/hyperframes): 0.43 without to 0.98 with on Sonnet, 0.62 to 0.97 on Opus.
- prototype (mattpocock/skills): 0.36 to 0.93 on Opus.
- triage (mattpocock/skills): a perfect 1.00 with the skill on both models, against about 0.7 without.
The ones we couldn't call
10 of the 40 results came back "too few answers to tell". grill-me, tdd and Anthropic's own frontend-design landed there on both models.
That is not "no effect". It means answers varied too much from one draw to the next to say either way, even after up to 10 draws per arm. We'd rather print that than a number that flips on a re-run.
How it works
- For each skill, three text tasks written from the skill's own description and README, fixed before any run.
- Each task is answered with and without the skill, 3 to 10 times per arm, and each answer is scored 5 times.
- "Clearly helped" means that on at least one task the two score ranges (mean plus or minus one standard deviation) don't overlap and the gap is at least 0.05, and no task went clearly the other way. A clear drop on any task wins over a gain elsewhere.
Limits
- Three text-only tasks per skill, written by us, not by the authors. No tool calls, generated media or deployed services were tested.
- One judge model.
- Popularity here is reported installs, not quality.
If you think a task is unfair or a result is wrong, every skill page has a link to ask for a re-run, and every receipt, task suite and retained answer is public.
- Full table, one page per skill: https://driftproofhq.com/skills/
- Report 015: https://driftproofhq.com/reports/015/
Disclosure: I maintain Driftproof, the open-source tool that produced these results.
Top comments (5)
The "too few answers to tell" bucket looks more informative than the headline. grill-me, tdd and frontend-design landed there on both models, and what those three change is a sequence of actions rather than the text of one answer. find-skills is the same kind of thing and it landed in "clearly hurt", which you already flag by saying you tested its written answer and not the skills it hands off to. So a single-answer score may be mis-sorting one class of skill in two directions at once, not just failing to resolve it.
That makes the undecidable bucket a category rather than noise, and it is testable. For those skills, score a property of the transcript instead of the quality of the prose: did it ask a clarifying question before producing anything, did a failing test appear before the implementation, how many steps before the first write. Binary or count outcomes over the same draw budget should separate far more cleanly than a 0 to 1 quality score, and if they do not, that is a real finding about the skill.
Worth saying too that your asymmetry rule, a clear drop anywhere beating a gain elsewhere, is strict in a way that structurally penalises any skill whose job is to route rather than to answer.
Fair, and you're right on both counts.
On the routers: the report already says that for find-skills, agent-browser, hyperframes and hyperframes-registry we scored the router's written reply, not the skill it hands off to. Given that, giving them a verdict at all was a mistake, in either direction. We're taking all four out of the verdict counts (the ones that helped as well as the ones that hurt) and marking them "not measurable with text tasks". The receipts stay as recorded. That leaves 32 results: 21 helped, 3 hurt, 8 too few to tell. It also means the headline on this post overstates it: on these tasks we can't say find-skills made anything worse.
On the asymmetry: it's deliberate for skills whose answer is the product, because whoever hits the task that dropped pays for it, whatever happened elsewhere. For something whose job is to route, it's the wrong rule, which is the other reason to take routers out of verdicts rather than score them leniently.
On grill-me and tdd: agreed, transcript properties are the better test. Receipts can already carry deterministic post-checks; we just didn't define any for these suites. "Asked before producing anything" and "failing test before the implementation" are both binary checks we could run on the answers we kept, before spending on new draws. If you were writing the tdd check, what would you count as a pass?
Installs measure popularity, not benefit. A skill adds context to each call, so it has to win on pass rate and on tokens per solved task, with it on and off.
Agreed. Cost next to lift is on our list for exactly that reason: a skill that wins by 0.05 while doubling the context is a different result from one that wins by 0.05 for free. Would you want that per task, or as tokens per pass at a fixed score threshold?
@driftproofhq Per task, with the pass count printed next to it. That keeps a setup that fails more from looking cheaper.