DEV Community

Cover image for Who's Auditing the Skills Your AI Agent Runs?
sunnydachs
sunnydachs

Posted on Edited on

Who's Auditing the Skills Your AI Agent Runs?

A lot of people have recently started installing "skills" into Claude Code, Codex, and similar agents. Skills are convenient — but behind that convenience, a pile of assets nobody is inspecting keeps growing.

So I built skilltrust, a CLI that inventories your installed skills in one command. In the first half of this post I'll explain why "read everything before you use it" is not a defense, and in the second half I'll show the CLI and its design.

GitHub logo sunnydachs / skilltrust

Deterministic, offline, stdlib-only CLI that audits an AI agent's trust surface (skills, hooks, trusted dirs, gh token scopes)

skilltrust

Audit your AI-agent trust surface in one command: skill inventories, trust settings, and token scopes — deterministic, offline, stdlib-only. It never judges skill content; it inventories facts so you can decide by provenance and permissions.

English | 日本語

License: MIT Python Tests CI

Why it exists

Agent skills (Claude Code SKILL.md, OpenClaw) are prompt-payload packages: a skill's text IS an instruction to your agent, inherited alongside your shell, filesystem, and stored credentials. Research in 2026 quantified the risk — the study behind NVIDIA's SkillSpector ("Agent Skills in the Wild", Liu et al.) analyzed 31,132 of 42,447 collected skills and found 26.1% carry at least one vulnerability and 5.2% show likely malicious intent; the USENIX Security '26 paper "Do Not Mention This to the User" behaviorally confirmed 157 of 98,380 analyzed skills as malicious. Snyk's ToxicSkills scan found 36.8% of 3,984 marketplace skills defective, 76 of them outright malicious payloads. An audit of the…

What a skill actually is: a package that runs with your permissions

An agent skill is a package that ships the prompt itself. Once installed, it becomes part of your agent's reasoning, and when it runs it touches your shell and your files — it inherits, exactly, the permissions you granted your agent.

If a skill's contents get swapped, it becomes someone else's code running on your machine.
Someone else's code that reads your files, can reach your credentials, and runs without asking you.

The question stops being "I installed it from a trusted source, so it's fine" and becomes "what is on this machine right now?"

This is not theory — it has been measured

Not a scare story: 2026 research and audits have the numbers. The foundation study behind NVIDIA's SkillSpector collected 42,447 skills and analyzed 31,132 of them: 26.1% contained at least one vulnerability, and 5.2% showed high-severity patterns suggesting malicious intent.

The USENIX Security '26 paper analyzed 98,380 skills and confirmed 157 of them as clearly malicious — deliberate bundles of 632 vulnerabilities across 13 attack techniques. Snyk's ToxicSkills report found defects in 36.8% of 3,984 marketplace skills, 76 of them outright malicious payloads.

In the ClawHavoc campaign, an audit alone found 341 malicious skills — most distributing a credential stealer called Atomic Stealer.

In other words: skills that are dangerous before anyone reads them are already circulating, in volume, as installed inventory.

"Just read every skill before you use it" — why that doesn't work

At this point the standard advice appears: read every skill before you use it. That advice has two practical limits.

First, it doesn't scale — reviewing every skill in a pile that keeps growing is impossible for one person. Second, and this is the deeper problem:

The text you are reading can itself be the payload.

A sentence that steers your agent can sit inside instructions that read as completely safe, and the human eye will not catch it. Static scanners share the same weakness against intent written in natural language.

So what is an effective defense? My answer: know your provenance and blast radius. Concretely, four things:

  • What is installed
  • Which directories you have marked as "trusted"
  • Whether hooks (commands that fire when a session starts) are enabled
  • Whether over-broad GitHub tokens are reachable from your agent

skilltrust: that inventory in one command

skilltrust turns those four questions into one command. What it does inside:

Installed skills        ──┐
Trusted directories     ──┼→ RiskMarkers ──→ ranked checks ──→ one action
GitHub token scopes     ──┤  (derived only)    ↑ first hit      (just one)
Claude hooks on/off     ──┘
Enter fullscreen mode Exit fullscreen mode

It gathers the four inputs, boils them down to derived values called RiskMarkers, walks a fixed priority ladder, and prints exactly one line: what to do next.

A sample run (from the README; counts vary by machine):

AI-agent trust-surface scan — 2026-10-09

[Skill inventory]
  Claude Code : 12 skills (2 script-bundled)
  .agents     : 12 skills (2 script-bundled)

[Trust settings]
  Trusted project dirs: 1 (/home/you/projects/example)
  Claude hooks: no

[Tokens]
  gh: 'workflow', 'delete_repo'  ← workflow, delete_repo overbroad

[Do this one thing]
  → narrow the gh token scope (5 min)
Enter fullscreen mode Exit fullscreen mode

Note: .agents is the convention for a skill location shared across agents.

The only part you need to read is the last section: the single highest-priority action, in one line. In this example it suggests narrowing an over-broad GitHub token scope — a five-minute fix.

A workflow-scoped token can rewrite CI and read the secrets of every repository you own, so it is worth killing first. The full sample output is in the README.

Why not let an LLM judge the contents

By now, someone is thinking: why not just have an LLM decide whether a skill's contents are dangerous? I deliberately don't. Two reasons.

First, an LLM's content verdict cannot be called trustworthy. As covered above, concluding "this text is safe" from reading the text is itself attackable — and if the judge can't be trusted, you can't make decisions on its output.

Facts — provenance and permissions — are the opposite: you can act on them.

Second, determinism has operational value. With the criteria fixed in code, no LLM and no network calls, the same environment produces the same result every time — so you can run it casually at install time, on pull, or any moment something might have changed.

Honest limitations

  • It does not judge whether skill contents are dangerous. It looks only at the presence and count of bundled scripts — a marker with a measured 2.12x higher vulnerability rate.
  • If a supported agent changes its config file format, skilltrust needs a tracking update.
  • Without the gh command installed, the token section reports "unavailable".
  • It never prints token values. Scope names only.

What it's like to run (measured)

I ran it on WSL2 with Python 3.14, scanning an ordinary personal dev environment with a handful of config files. What that looks like, with measured numbers:

  • Runtime: 0.01 s (three measurements — it feels instant)
  • Tests: 23, all passing (CI runs the same set on Python 3.11–3.14)
  • Implementation: 814 lines, zero dependencies (Python standard library only)

All it does is read a few config files, so it should finish within seconds on your machine too.

Summary: inventory before you read

Skill safety is not decided by "did I read all of it" — it is decided by whether you know your provenance and blast radius. skilltrust does that check in one command.

Knowing where to look first when something breaks matters. But the bigger value is keeping the inventory current before anything breaks — if it is always up to date, you notice the moment it differs from your usual list.

Code: skilltrust on GitHub — bug reports and ideas welcome in issues.

This is a personal OSS project with no warranty. If you hit bugs or have suggestions, GitHub issues are the best way to reach me.

Top comments (4)

Collapse
 
mayailands profile image
Maya •

Three ways the report reads clean when it has not looked. skilltrust 0.1.0, commit 1b57680. All three ran offline against a fake HOME, no network.

1) The hooks line contradicts its own action

Fixture: ~/.claude/settings.json with a real hooks key.

[Trust settings]
  Claude hooks: no

[Do this one thing]
  → review ~/.claude/settings.json hooks
Enter fullscreen mode Exit fullscreen mode

Same run, --json: trust.claude_hooks_present=false, risk_markers.hooks_present=true, suggested_action="review ~/.claude/settings.json hooks". Cause: get_hermes_trust_settings hardcodes claude_hooks_present=False ("Will be set separately"), the CLI hands the real value only to compute_risk_markers, and format_human_output prints the trust field. The display leg is never wired. test_cli feeds claude_hooks_present=True straight into the result, so the wiring is the one path the tests do not walk.

2) A valid trust list reads as zero

Same machine, same four entries, two spellings.

skills:
  trusted_project_dirs:
    - /proj/one          # documented block form
Enter fullscreen mode Exit fullscreen mode

→ Trusted project dirs: 4

skills:
  trusted_project_dirs:
  - /proj/one            # items at the key's own indent, also valid YAML
Enter fullscreen mode Exit fullscreen mode

→ Trusted project dirs: 0 (none), hermes_parse_error: false

The flow form trusted_project_dirs: [/proj/one, /proj/two] also reads 0. Cause: parse_hermes_config is a hand-rolled subset that accepts 4-space block items only, and everything else returns an empty list with no error. So trust_list_size=0, and the priority-4 action ("prune trusted_project_dirs") can never fire on a machine with any trust list. "Zero trusted dirs" and "I could not read your trust list" print the same line.

3) A hook you cannot read reads as no hook

Same file, valid JSON → hooks marker true, action = review the hooks. Truncate the same file mid-object → get_claude_hooks_present swallows the JSONDecodeError and returns False, and the action falls through to the next priority. The unreadable case and the no-hooks case share one marker.

Layer. Part of this is unavoidable: the files are written by the same party whose trust you are grading, so the report has to read them and say what it found. The cheap half is not unavoidable. Nothing in the output distinguishes "checked and absent" from "could not read", and so every failed read silently becomes a clean negative. Carry a tri-state per leg (present / absent / unreadable), print not checked (reason) instead of a negative, and let the clean no action needed this cycle be refused while any leg is unchecked. That is what turns priority 1-4 into a ladder instead of a fall-through.

Control that must pass: documented block form plus valid settings.json → 4 dirs, "review hooks". It does.
Negative control that must fail: the same truth spelled as above → must not print 0 and clean. It does.

The finding is free; that is what coordinated disclosure means. If you fix it, the re-run receipt (this format, run against the fixed build, quotable in your release notes) is one check, $25, 48h. Say the word and I'll send the card link.

One thing on channel: SECURITY.md asks for private reports and I could not use that door, I don't have a GitHub account. If this class should stay private, tell me and I will route it that way next time.

Collapse
 
sunnydachs profile image
sunnydachs •

Thank you for the report - all three reproduce on 1b57680, root causes where you put them, and all three are fixed in 501fa0d, in your tri-state shape: present, absent and unreadable are separate findings, 'not checked (reason)' replaces every silent negative, and the clean verdict is refused while any leg is unreadable. The key-own-indent spelling now reads the same four dirs as the documented one, and the hooks row agrees with the action it produced. The findings were the valuable half and free - no card needed. On the channel: this class is fine public - a false negative in a config report is the worst case, and they're public already.

Collapse
 
mayailands profile image
Maya •

Verified on 501fa0d, offline, fake HOME, no network. All three original cells, with controls.

  1. ~/.claude/settings.json with a real hooks key → Claude hooks: yes, and the action agrees with it. Control (valid JSON, no hooks key) → no + no action needed this cycle.
  2. The same four dirs at the key's own indent → 4 (/proj/one, /proj/two, /proj/three, /proj/four), same as the documented block form.
  3. Truncated settings.json → Claude hooks: not checked (unreadable settings.json) and the action is resolve the not checked rows before trusting a clean scan. The clean verdict is refused, which is the part that matters.

The old commit still fails the same three: Claude hooks: no printed next to → review ~/.claude/settings.json hooks (1), 0 for the key's-own-indent list (2), no action needed this cycle on the truncated file (3). 32 tests pass on 501fa0d.

One leg left, same root one layer out. The legacy JSON booleans collapse the tri-state you just added: on the truncated file, trust.claude_hooks_present and risk_markers.hooks_present both read false while claude_hooks_state is unreadable. unreadable_legs carries the truth, so a consumer reading the state fields is fine, but one reading the boolean alone gets the old silent negative. null when the state is unreadable closes it. A note, not a fourth bug.

Fix verified. The re-run is yours to quote if a changelog line wants it.

Thread Thread
 
sunnydachs profile image
sunnydachs •

Thanks for the re-run - the cells and both controls match what I measure here. The note is right and it is on us: the fix swept the human rows and the state fields but left the legacy JSON booleans two-valued, so a boolean-only reader still got the silent negative. Fixed in 68456b4, taken wider: every JSON field that means nothing on an unreadable leg - the two booleans, the counts, the path lists - is null now, with tests pinning null-on-unreadable and false-on-absent. The re-run quote is appreciated.