An agent calling your MCP server never sees your code. It sees tool names, descriptions and parameter docs, and decides from those alone which tool to call and what to send. When a description is missing or wrong, the agent picks the wrong tool or sends bad arguments, and nothing in your own tests notices.
I built mcp-doctor to check for that. It's a static linter for MCP server source (Python, TypeScript, Go) that grades quality and security separately. It never starts the server. This week it also ships as a Claude Code plugin, and this post is about the part I spent the most time on: testing that the plugin doesn't make things up.
Why a plugin, not just the CLI
The CLI is good at the mechanical half: finding tools, spotting a parameter with no description, an optional argument typed non-null, a print() on a stdio server. It can't write a good tool description, because that needs someone to read what the tool actually does.
An agent can do that half. So the plugin is a skill that:
- runs the CLI and reports both grades plus at most five findings, each phrased as what goes wrong for the agent ("the model can't tell when to call
run"), not as a rule name; - shows the mechanical fixes as a diff and applies them when you say yes;
- reads each tool's code and writes the descriptions, parameter docs and error messages the CLI can't;
- re-scans and shows the grades before and after.
Install:
/plugin install mcp-doctor --marketplace vishalhabib99/mcp-doctor
Then ask "check my MCP server".
The risk: a confident, wrong description
Handing description-writing to a model is risky in a specific way. A model asked to "add a description" to a tool called run that takes a cmd argument will happily write "Executes a shell command". That reads well and could be completely false.
The sample server in the repo has exactly that tool. run(cmd) ignores cmd and returns the string "ran". A false description there is worse than none: it tells the next agent this tool can run commands, so the agent will try.
So the skill's instructions say: read the tool's code before writing anything, and never describe behavior the code doesn't have. Instructions are cheap, though. I wanted a test that fails if the model ignores them.
The eval suite
The plugin ships with five cases, run with claude plugin eval:
| Case | What it checks |
|---|---|
scan-bad-server |
The skill fires, runs the CLI, reports Quality 13% F and Security 100% A separately, edits nothing |
fix-bad-server |
Applies fixes, shows before and after, and does not describe run as executing a command |
clean-good-server |
Reports A / A on a clean server and proposes no fixes |
not-mcp-code |
Doesn't fire on unrelated Python work |
mcp-config-not-server |
Doesn't fire when you want to use an MCP server, not build one |
The last two matter as much as the first three. A skill that fires on every Python file is noise, and one that "fixes" a clean server can't be trusted.
There's also a rule for when the CLI can't run (not installed, no network, Bash not allowed): say so in the first line and never state a grade the CLI didn't produce. The eval sandbox blocks the installed CLI, so that path got tested whether I liked it or not. The graders accept either the real grades or a plain "the scanner didn't run" with no invented number.
Results
Last full run: 3 runs per case, with and without the plugin.
- The skill fired in 9 of 9 MCP runs and 0 of 6 unrelated ones.
- 13 of 15 runs passed every grader. Overall score was 0.94 with the plugin, 0.63 without it.
- Both misses were judge votes on runs where the sandbox blocked the CLI and the skill said so without inventing a grade. I count that as the honesty rule working and the judge being strict, but they're still misses.
In a normal session with the CLI available, the skill took the sample server from 13% to 42%. The rest of the gap is a missing README, LICENSE, tests and pinned dependencies, which it listed but didn't create, because a LICENSE is your call, not the agent's. It described run as a placeholder that ignores cmd. On the clean server it reported A / A and changed nothing.
What's underneath
The plugin is only as good as the scanner. The CLI finds 15,755 of 15,977 tools across 805 public servers, with every miss listed in coverage.md. Running it on real servers has turned up 104 bugs in mcp-doctor itself, all fixed, and 22 fixes I've sent to other people's MCP servers have been merged.
Takeaway
If you ship an agent skill that writes anything a later agent will rely on, test the failure where it writes something plausible and false. Put a tool in your fixtures whose name invites a wrong description, and fail the run when the model takes the bait.
Bug reports are welcome, especially servers where it gets the tool count wrong: github.com/vishalhabib99/mcp-doctor
Top comments (6)
Re-deriving the table: 3 MCP cases x 3 runs = 9 fired, 2 unrelated cases x 3 = 6 not fired, and 13 of 15 is 87% fully passing. With 3 runs per case, one flaky run moves a case from 100% to 67%, so I would read 0.94 vs 0.63 as "clearly better" rather than as a precise gap. Also, the two misses were both on the CLI-blocked path, which is the one case where the judge, not a deterministic grader, decides; a grader that checks "no grade number appears in the output unless it matches the CLI's output" would make that path pass or fail without a judge.
A static linter grades the description text, and there is one failure it cannot see: a description that is accurate and still loses when the agent picks a tool. In our own tool search over 306 tools, "silver price" returned US CPI and the stemmer turned "rates" into "rat". Both were fixed by changing the lookup, not the descriptions. Do you have a case in the suite where two well-described tools are near-duplicates and the agent has to pick the right one?
I built canli-mcp (free, MIT, no key,
npx -y canli-mcp, github.com/arhancanli/canlicapital). If you want a large real server to run mcp-doctor on, it is one; I would be curious which findings it reports.Your re-derivation is right, and I'll take the correction: with 3 runs per case, 0.94 vs 0.63 says "clearly better," not a measured 0.31 gap. The CLI-blocked path is a good catch too. A deterministic grader ("any grade in the output must match the CLI's") would take the judge out of the one case where both misses happened.
On near-duplicates: no, the suite has no pick-between-two case. mcp-doctor only flags descriptions that are identical after normalizing. A similarity score flagged good CRUD families like update_person / update_company, so I dropped it. Your "silver price" → CPI case is exactly what it can't see, since the fix was in the lookup, not the text.
I ran it on canlicapital (bb7001c): Quality 89% B, Security 98% A. It reads 84 tool definitions behind your find/describe/run layer. Real findings: 11 Zod fields with no .describe() across 8 tools (pit_factor has 2 of 5 undocumented), and 33 tools not mentioned in the README. It also showed two bugs on my side: it reads your tool() helper's title ("MACD", "TRIX") as the description, and it counts same-named tools in separate servers of the monorepo as duplicates. I'll fix both. Thanks for the test case.
Both fixes shipped in v1.24.1 (PyPI
mcp-server-lint0.24.1). On canlicapital the duplicate-name warning goes from 17 names to 1, and the MACD/TRIX description findings are gone. The one left ischeck_orders, which the hosted build and the local build of mcp-execution both register in the same package. That may be intended, so it stays as a warning. Thanks again for the test case.Thanks for running it on a server this size. The two bugs it showed on your side (the helper's title read as the description, same-named tools in separate monorepo servers counted as duplicates) are what a big real server is good for, and the pit_factor fields are a fair finding; I've passed the list to whoever maintains it.
One thing that may be worth separating in the report: the 84 definitions sit behind a find/describe/run layer, so an agent only ever sees three front tools and reaches the rest through search. A quality score over the catalog tells you the descriptions are clean, but not whether the front three are clear enough to get the agent to the right entry. If mcp-doctor can grade the front tools on their own line, the 89% would say which of the two it is measuring.
Treating the plugin description as a public interface with tests is worth stealing, since the calling agent only ever reads names, descriptions and parameter fields. How do you decide a tool description has failed rather than the underlying tool, because those two look identical from the outside if the agent calls the wrong one. Do you keep a case where the description is fine and the model still misreads it, as the control that stops you rewriting prose forever? And once a description passes, does anything stop the next edit to the handler from invalidating the eval that guards it?
Good questions. In this suite the tool side is fixed: the test servers are scaffolded copies and the CLI is deterministic, so a wrong result points at the skill and its description, not the tool.
The controls are cases that must not fire: unrelated Python work, and a user who wants to use an MCP server rather than build one (0 of 6 fired), plus a clean server where it must propose no fixes. Those stop me rewriting prose just to make it fire more. I don't have a fixed "good description, model still misreads it" case. 3 runs per case is the only read on that today.
On regressions: honestly, nothing automatic. The CLI's grading is unit-tested in CI, but the skill eval is about 30 full Claude sessions, so it's a manual rerun, not a CI gate yet.