We published the scores of 55 public MCP servers, 1,274 tools in all, at forecall.dev/en/servers. Each tool description is scored out of 100 on whether it says what the tool is for, when to use it, its parameters, what it returns, its constraints and an example.
Then we tested whether a low score means models pick the wrong tool. For one-step choices, it didn't: two frontier models picked the right tool 91–99% of the time, low scores or not. This post covers both: what the descriptions leave out, and what that does and doesn't tell you.
Where the 1,274 tools came from
- 46 servers are Smithery's most-used servers, copied from its registry API on October 7, 2026 (UTC). The registry keeps each tool's name, description and input schema, but drops output schemas, annotations and server instructions.
- 9 servers were taken from the running server with
forecall dump: Notion, Playwright, Chrome DevTools, Context7, Filesystem, Memory, Firecrawl, DebugBase and Bhived.
All were scored with the same rules (Forecall's scoring rules v2). Each server's page says which source it came from and when. The pages show scores, issues and tool names, not the descriptions themselves. If you own one of these servers and want it taken down, email [email protected] and it's removed within 7 days.
What the descriptions leave out
885 of 1,274 tools (69%) never say when to use them. The share is the same in both groups: 69% of the registry copies and 71% of the dumped tools.
87 of the 156 dumped tools (56%) don't say what they return, neither in an output schema nor in the description. (I'm leaving the registry copies out of this one, because the registry drops output schemas.)
18 of the 55 servers have at least one pair of near-identical descriptions, 129 pairs in all. Two servers account for most of them: Notion (57 pairs across 24 tools) and Paper Search (29 across 25).
Server averages range from 19.9 to 82, with a median of 44. Across all tools, 497 score under 40 and only 18 score 80 or more.
Does a low score mean models choose wrong? We measured it
Forecall also runs evaluations: it writes requests for each tool and asks models which tool they'd call. We used that to calibrate the static score on 4 servers with 88 tools (Memory, Notion, Playwright and Chrome DevTools), 8 one-step requests per tool, with Claude Sonnet 5.5 and GPT-6.1 Sol:
| Server | Tools | Average score | Right tool (Sonnet / Sol) |
|---|---|---|---|
| Memory | 9 | 43.8 | 97% / 99% |
| Notion | 24 | 33.0 | 96% / 96% |
| Playwright | 25 | 41.9 | 93% / 94% |
| Chrome DevTools | 30 | 45.1 | 99% / 98% |
The correlation between a tool's score and how often models picked it correctly was 0.04 (Spearman). Tools scoring under 40 were picked correctly 95.5% of the time; tools scoring 60–79, 100%.
We then put 25–55 tools from other servers next to each one, to make the choice harder. None of the requests that needed a tool went to the other servers' tools, and accuracy stayed at 91–98%.
So, for frontier models choosing in one step, a short description wasn't the problem: the tool names and schemas carried enough. The misses that did happen were mostly the models doing something sensible first. When a request didn't include a page number, they called list_pages before acting, in 77–87% of those requests in an earlier run. For browser_fill_form, they took a page snapshot first, because the field references come from it.
What the score is for, then
The score says what a description puts in words, not how a model will choose. We changed the site's wording to match: what we used to call "pairs easy to mix up" are now "near-identical descriptions", and every result says what static scoring can't tell.
What's missing from the text may still matter where this test doesn't reach: when the user doesn't have the ID a tool needs, over several steps, for smaller models, and for the people reading your docs. Just don't assume a longer description is better. In an earlier test, rewrites that raised scores (one tool went from 31 to 74) made two models worse (details here). Check rewrites against the models your users run.
Check your own server
- Look it up: forecall.dev/en/servers
- Not listed? If it's a public Streamable HTTP server that needs no sign-in, paste its URL at forecall.dev/en/lint.
- Locally, with nothing sent anywhere:
npx forecall dump -o tools.json -- <command that starts your server>, thennpx forecall lint tools.json. The CLI also reads endpoint lists and search results. - To see how models actually pick your tools, Forecall's evaluations (Pro and Team plans) report the right-tool rate, look-ups first, argument validity and false calls per model.
I'm the author of Forecall. The linter is open source: github.com/forecall/forecall-cli.
Top comments (3)
The calibration has the model choose with every description already in its context, which is the step where a description has the least work to do. In Claude Code, where tool search is on by default, MCP tools are deferred and the model has to find a tool before it can choose it: the docs say the search matches Claude's query against each tool's description and advise putting "the words Claude would search for near the start". The API docs for tool search say both variants search names, descriptions, argument names and argument descriptions, return up to 5 tools by default, and list "Use keywords in descriptions that match how users describe tasks" among the tips, which is close to what a "when to use" sentence supplies.
So the cost of a low score, if there is one, would show before the choice: whether the right tool comes back from the search at all. The 25 to 55 added tools were told apart by a model that could see every one of them. Running the same requests with the tools deferred, and counting per score band how often the right tool is among those the search returns, would test the score at the step where the description is what gets matched.
You're right that the calibration only tests the step after retrieval: the model saw every tool, so a description only had to win the choice, not the search.
We have a rough proxy for the search step. Since this week, the evaluation page ranks the target tool with BM25 over the names and descriptions of the tools the model was shown, using the request as the query. On the same 88 tools, in a later run with the other servers' tools alongside (808 requests), how often the target landed in the top 5 didn't follow the score: 79.5% for tools under 40, 77.8% for 40–69 (Spearman 0.08). But that proxy differs from the real search in three ways: it leaves out argument names and descriptions, which both API variants search; the query is the user's request, not one Claude writes; and the requests were written by a model that had read the descriptions (it's told not to quote them), which can still favour a match. And only one tool scored 70 or more.
So we'll run it the way you describe: the same requests with the tools deferred behind the API's tool search (BM25 and regex), counting per score band how often the target comes back from the search, and how often it's then called. I'll post the numbers here.
Here are the numbers. Same 404 requests as before (88 tools of four servers, 25–55 tools of three other servers beside them), Claude Sonnet 5.5, every tool deferred behind the API's tool search, so the model started with the search tool only.
BM25: the right tool came back from one of the model's searches for 97.8% of requests (92.8% from the first search) and was called for 94.6%, against 96.5% with every tool shown. Regex: 92.1% came back (64.4% from the first search), 88.1% called.
By score band, came back / called:
Spearman between a tool's score and how often it came back: −0.06 (BM25), −0.08 (regex).
So at the step where the description is matched, the score didn't predict which tools get found either. The search did. Claude rewrote each request into its own query, so our earlier proxy over the user's words didn't track the real search (per-tool Spearman 0.04): it had browser_navigate outside the top 5 for all 4 of its requests, and Claude's searches found it every time. With regex, results came back in the order the server lists its tools (all 309 searches that returned two or more of a server's own tools), so a broad pattern like
database|dbon Notion's 24 tools stopped before API-retrieve-a-database. Most BM25 misses were Notion's API-get-user, outranked by other servers' tools with long descriptions and argument text.We've taken the proxy off the evaluation page. Not measured: smaller models, and Claude Code's own search, which also shows tool names up front.
A correction to my earlier reply: those were 404 requests answered by two models, not 808 requests; the percentages don't change.