I cut 41 AI tools' self-descriptions in half, then A/B tested that trigger rate didn't drop

The result first

Two more tools, 10,400 fewer characters. And the trigger rate didn't drop.

(±2 cases at this sample size is noise, so I am not claiming it got better. What I can claim: cut 40% of the characters, no systematic decline.)

If you already have dozens of AI tools installed, the first half of this is worth reading — because some of yours may not be loaded at all right now, and nothing will tell you.

The setup: a failure with no error message

Claude Code's skill mechanism works like this. Each skill is a folder with a SKILL.md, and the file opens with frontmatter:

---
name: atomic-db-operations
description: Use when writing a DB RPC that touches stock/balance/order state...
---

At session start, every skill's name + description gets packed into the model's system prompt. That listing is the only way it knows what tools it has. It can't see your folders. It can't read the files. It sees the list.

And that listing has a character budget.

When you exceed it, something very specific happens: the description is dropped and the name is kept.

So that skill:

• is still on the list, and /skill-name still invokes it manually ✅
• but it will never surface on its own again ❌ — the model sees a name and has no idea what it does or when to reach for it

No error. No warning. It just quietly stops showing up.

I'd written before about a related blind spot — tools that never got loaded at all because of the load path. This is the second layer of the same problem: the thing loaded, but its self-introduction was thrown away in transit.

How big is the budget?

I decompiled the local binary to check, because the community blog posts saying otherwise were second-hand and wrong. The actual rule:

• Budget = 1% of the context window (a setting called skillListingBudgetFraction, default 0.01)
• A 200K-token window ≈ 8,000 characters; a 1M window ≈ 40,000
• There's also a per-skill cap (skillListingMaxDescChars, default 1536)
• The env var SLASH_COMMAND_TOOL_CHAR_BUDGET overrides the whole calculation

What I measured at the time: 89 project skills ≈ 22,700 chars + 23 global skills ≈ 11,700 = about 34,400 characters, against a default budget of 8,000.

The way to see it is blunt: in my available-skills listing, roughly 70 entries were name-only.

Why this gets worse on its own

The docs spell out the eviction order: "starting with your least-used skills."

That reads as reasonable — keep the popular ones. It's actually a death spiral:

never used → description evicted → model can't see it → used even less → evicted first again

I'd separately measured my toolbox's usage rate: 79 of 89 had never been explicitly invoked. I wrote that up as its own post at the time. Looking back, that number was contaminated by two things at once:

1. genuinely not useful
2. budget overflow, so the model was never told it existed

Until you fix the second, the first number can't justify retiring anything. That's the mistake I nearly made — I was about to delete a batch of skills based on the "zero usage" list.

The decision: trim, don't cut

First I raised the budget (set that env var to 45000, in two config files), and verified end to end: opened a headless sub-session and forced it to quote verbatim the description of a skill that had been name-only. It quoted it, and the words matched the file.

Two weeks later the budget was full again (skills kept accumulating). This time 8 were stripped to names.

So the question became: cut tools, or trim descriptions?

I did the arithmetic. Squeezing every description under 250 characters saves 8,058 characters. Deleting all 51 "tier C" skills (the least-used batch) saves 11,295.

Trimming recovers nearly as much as deleting half the toolbox — and retires nothing.

That's why I trimmed. Not aesthetics. Arithmetic.

The hard part: how do I know I didn't cut a trigger word

This is the real technical risk.

That description is the basis on which the model decides whether to invoke the skill. Cut a trigger word and it silently stops firing in that situation — and you won't know. No error. It just never appears.

My first approach was "delete the redundant-feeling parts." The problem with that approach: I had no way to prove to myself that I hadn't cut something load-bearing.

So I changed to three steps.

Step 1: only trim three categories, touch nothing else

• The same sentence written twice in two languages (e.g. a Chinese phrase followed by adding/renaming a field across write→middleware→render)
• e.g. enumerations (handleUpload + handleBatchUpload + handleDragDrop + handleUrlInput → keep three)
• Restatements of the SKILL.md body (the description doesn't need to re-explain what the five steps are; those words are in the file)

Trigger words — identifiers, symptom phrases, SKIP routing to other skills — untouched.

Step 2: mechanical reconciliation, not self-report

After trimming, run a script: split the old and new versions each into two sets — "ASCII identifiers" and "natural-language fragments" — then print every token present in the old and absent in the new.

The point of this step isn't automation. It's turning "did I cut a trigger word?" from an internal state only I can see into a list lying on the table. I can't reassure myself past a list.

Result: all 24 had token loss. Reading through, most were function words (the, and, that, use, when) and things I'd judged as restatement — but 8 had genuinely lost trigger words:

• label map, trigger (two of four load-bearing positions in one rule)
• isInQuietHours, deferred:true (code-level signals — anyone reading or writing these symbols should trigger this skill)
• line-auth (the name of an Edge Function)
• a natural phrasing for "change the unit price" — I'd kept only the identifier single_price
• a full sentence a client actually says: "I uploaded the images, please place them for me"
• two SKIP routes pointing at other skills

All restored. Without that list I would not have caught a single one of the eight.

Step 3: two stale claims fell out for free

Reconciliation forces you to read the old version word by word, so I picked up two errors unrelated to trimming:

• one description said "three-part report" while its own SKILL.md and my rules file both say four-part (the "three" referred to three markers inside part 3, misread as the whole report's structure)
• another described the precedence of a fallback mechanism that had been removed in a database migration months earlier

Reading word by word catches these. Skimming doesn't.

Then I needed behavioral evidence

At this point I had two pieces of evidence:

1. every removed token reviewed line by line (mechanical, but indirect — it proves the words are still there, not that the skill still fires)
2. after a system-prompt reload, all 24 new descriptions appeared verbatim (proves delivery, not trigger rate)

Neither is behavioral evidence. And I happened to have a question bank: 4 test cases per skill (should-trigger, strict variant, should-not-trigger, boundary), 96 total — run against a cheap model, three votes per case, majority wins.

(An aside: my handoff doc claimed "most of these 24 have no test cases." That sentence was wrong; all 24 had them. How I caught that wrong sentence is a story of its own, and I wrote it up separately: my test report printed "0/96, 0% pass rate" — the truth was my account was out of credit.)

Why "94% after" isn't evidence

The run came back 90/96 (94%). Looks fine.

But it can't answer the question I'm asking. The question is "did it drop?", and 94% is an absolute number — I have no "before" to compare it to.

So I opened a working copy of the old version (git worktree pointed at the pre-trim commit) and ran the same question bank, the same model, the same day, the same machine. The only difference was the length of those 24 descriptions.

288 API calls per arm, about US$0.32 each.

A/B result

Only 4 skills moved at all, each by exactly one case:

The other 20 did not move.

The honest reading

I will not say "trimming improved triggering." Net +2 cases, 96 cases total, one round per arm — that's inside the noise. Three up and one down looks like jitter, not a trend.

Here is what I can claim: cut the total characters across those 24 descriptions by 33% (8,407 → 5,653), and behavioral tests showed no systematic decline.

The one to watch is the one that fell — data-contract-propagation-audit, which also happens to be the one I squeezed hardest (689 → 247, −64%). It lost one case. If it's still down next round, that's genuinely over-trimmed and it should get more of the character budget back.

There's a practical lesson buried here: compression ratio correlates with risk. Everything I cut by 30% held steady. The only one that moved was the one I cut by 64%.

The global batch: a trap that almost wasted the whole exercise

After the 24 project skills, I still had 17 "global" skills (shared across all projects). That batch had an extra problem: some of them have their source of truth in a different repo.

Edit the wrong copy and the next sync overwrites it — the entire effort gone.

So I checked where each one's source lived. My first method was "find the folder whose name matches the skill name."

That method was wrong. One skill's source file sits in a folder that is not named after it. I came very close to editing a copy that gets overwritten on the next install.

The correct key is the name: field written inside the file, not the folder name. Re-running the classification with that key sorted it out.

And I made a second mistake: when I built the work list, I copied the 9 skills listed in my own previous report — and missed one. It had always been in that group; my earlier report just hadn't listed them all.

What caught it was re-measuring the over-budget list after applying the changes — it was still on there.

Working from your own last report ≠ working from a fresh measurement.
Neither mistake was technical. Both were "I used a proxy that looked good enough."

Net result

Not one tool was retired.

Five things you can take away

1. First check whether your listing overflows. Sum the description lengths across all your SKILL.md files and compare to "context window × 1%." If you're over, some of your tools are name-only right now.

2. The fix for overflow is trimming, not deleting. I did the math: squeezing to 250 chars saves as much as deleting more than half the toolbox. And before you delete anything — that "zero usage" list may be contaminated by the overflow itself.

3. Classify before you cut, reconcile mechanically after. Only trim bilingual restatement, e.g. enumerations, and body-text repetition — then print every token that existed before and doesn't now, and judge them one at a time. That's how I recovered 8 wrongly-cut trigger words. Without the list I'd have caught zero.

4. An absolute number can't answer "did it drop." 94% is a nice number and it is not evidence. A working copy of the old version running the same bank is. The extra cost was one round of API spend (US$0.32 here) in exchange for a sentence you can actually stand on.

5. The harder you compress, the closer you watch. Everything I cut 30% held. The only regression was the one I cut 64%.

Postscript: use the right key to find the source of truth

If you also have "the same skill exists in several repos," find the source using the name: field in the frontmatter, not the folder name. I have a separate audit script that compares content hashes across copies — after this change I ran it and confirmed the drift set was identical to before, zero additions. That is evidence the copies are still in sync, as opposed to my feeling that they ought to be.

This whole exercise is one layer of what I've elsewhere called the harness — the shell you wrap around the model, where the part only you have lives. If you want the wider version of that argument, I wrote it up here: NVIDIA's CEO says future companies will be built on harness engineering.

Originally published on my blog: I cut 41 AI tools' self-descriptions in half, then A/B tested that trigger rate didn't drop

I keep a running index of every pothole I've hit building a real production system solo — symptom on the left, what to grep in your own repo on the right: coffeeshooters.com/potholes

And if your team is shipping AI-written code faster than anyone can read it, that's the thing I do for a living: coffeeshooters.com/code-audit