I run ProtocolProbe, a directory of facts about every server in the official MCP Registry: each server gets a page, read from public records (the registry, npm, PyPI, GitHub), with a date on every fact.
A publisher emailed us. They had renamed their server, marked the old registry entry as deprecated, and added a message pointing to the new name. Our page for the old entry still said active.
I told them it would update on our next daily run. That was wrong. This post is about why, and what we changed.
The bug
Our daily job pulls the full registry list, compares it with yesterday's, and loads what changed: new servers, new versions, removed servers. For those, it re-reads the package and repository facts the same day.
The publisher's change was none of those. The registry lets a publisher change an entry's status (active → deprecated) and its status message without publishing a new version. Same version, different status.
Our job noticed it: it counted it as a "metadata-only" change. It just never loaded it. The page kept the old status, and the run still finished with ok.
How big was it?
One report can be a coincidence, so we compared every stored record with the registry list of that day, field by field (read-only, 51,834 servers):
| mismatches | |
|---|---|
| servers missing / extra | 0 / 0 |
| version | 0 |
status (stored active, registry deprecated) |
53 |
| status message | 60 |
So the parts the job was built for were exact, and the gap was precisely the case it skipped: entries deprecated by their publishers that we still showed as active. Our own record check "deprecated in the registry" gave the wrong answer for all of them.
Fix 1: reconcile what the registry owns, every day
The registry is the source of truth for a few fields: status, status message, published and updated dates. Now, after every daily load, the job compares those fields for all servers with the registry list and patches the ones that differ at the same version:
- a backup of every value before the first write, and a journal, so a crash mid-way resumes instead of half-patching;
- every patched document read back; a mismatch rolls the whole step back;
- a cap of 2,000 patches a day (a registry-wide change needs a person to look);
- each change recorded publicly with its date, so anyone can see what changed and when.
First run: 59 servers patched (some had changed since the count above). A second comparison right after: 0 mismatches across 52,138 servers.
Fix 2: a job that says "ok" must check the page
The deeper problem was not the missing field. It was that the job reported success without looking at what visitors see.
So the job now ends with a live check: after the pages are refreshed, it takes a small fixed sample of the day's changed servers (new, new version, status patched) and opens their public pages.
We first tried reading our own public API instead. It was the wrong thing to check, for two reasons we found by reading our code:
- The API is not what was stale. The page was.
- The API answer is cached at the CDN and inside each function instance, and the instance copy comes back without a cache header. A check right after a load can read an old copy and raise a false alarm.
Each server page now carries a small machine-readable marker in its head: version, status, a short hash of the status message (never the message itself), and the snapshot date. The job compares that with what it loaded.
- A mismatch is a warning, never a rollback.
- Our own failures (rate limits, timeouts) are "not checked", never a mismatch.
- If half the sample can't be checked, that is a warning too.
Fix 3: a date range, not a date
While checking this, we found a quieter version of the same problem on our leaderboards. They said "facts checked 2026-10-29". That was the newest date among the records, which meant the servers that changed that day.
Most stars and download counts behind the ranking had been read days earlier. The page now shows the range: "facts checked 2026-10-24 to 2026-10-29".
How stale does a directory get?
We measured it. We re-read the free facts (npm/PyPI, OSV, GitHub) of 1,729 servers that had not changed in the registry, a 1/30 slice of the catalogue. It took about 14 minutes and about 1,200 GitHub API requests.
614 of them (35%) had at least one fact that had changed since our last read:
- 396 download counts
- 301 last-push dates
- 99 star counts
- 23 version-pin states
- 7 repository states (not found, found again, archived)
- 1 newly withdrawn package
None of these show up as a registry change. We built a daily rotation that re-reads such a slice every day, so every server is re-read at least monthly. It is switched off until we have watched a few normal runs of the new checks.
What I'd tell anyone building on a registry
- A listing is a snapshot. The official MCP Registry checks an entry when it is published (each new version), and nothing re-checks it on a schedule. We asked for that upstream: registry issue #1715. If you mirror it, you inherit the same gap.
- Same version doesn't mean same record. Compare the fields the source owns, not just the version.
-
Check the output, not the exit code. Our job said
okfor days while pages were wrong. - Show the age of your data honestly. A single date can quietly overstate how fresh it is.
- Say when you were wrong. I told the publisher the wrong thing first. Admitting it took one email. Our corrections are logged at protocolprobe.com/corrections.
Every change the job records is public, by day, at protocolprobe.com/mcp/changes, and in a free API with no key (/docs/api). If you run an MCP server and something on its page looks wrong, /operators says how to tell us.
Top comments (17)
The 53 deprecated entries still showing as active is a useful find, and showing a date range on the leaderboard is a nice honest touch. The same version, different record problem goes one level deeper for remote servers: the endpoint can change what tools/list returns with no new version and no registry change at all. Do you record anything about the tool list itself, like a hash per server, or only the registry and package facts?
Yes, the tool list itself. Every read-only listing we capture is stored as hashes: one for the whole list, and one per tool for its title, description, input schema and annotations. Each server's reads are compared in order, and the hosted endpoint and the package are tracked separately so they're never mixed up. The server page then shows tools added, removed or changed between two reads, and flags "changed without a new version". When we first measured it on 2026-10-25, 0 of the changed pairs had the same version, so we haven't caught that case yet, but it's flagged when it happens. We never call the tools, and a read we don't have counts as unknown, never as "unchanged".
Counting a missing read as unknown, not unchanged, is the part I like most. That's the honest default. And 0 same-version changes so far is still a useful number to publish, it tells people the check is armed. Will keep an eye on the server pages.
Thanks, glad the unknown-vs-unchanged choice reads right. One caution on the 0: it is out of only 11 same-version pairs we could compare (as of 2026-10-31), so it shows the check ran and found nothing there, not that silent swaps don't happen. Our daily job only re-reads new or new-version listings, so a same-version change is mostly invisible to us today. We state that on the site as a coverage limit.
Stating the 11-pair limit on the site is the honest version, thanks for spelling it out. If you ever widen it, a slow weekly re-read of a random sample of unchanged listings would give you a real rate for same-version swaps instead of a blind spot.
Thanks, that's a good suggestion. You're right that today it's a blind spot: we only see a tool list on the days we read it, so a same-version swap shows up only if two of our reads happen to straddle it. A random sample of unchanged listings, re-read on a schedule, would turn that into a rate with a stated base. I'm noting it as the way to widen this. I won't promise a date, and I'd publish the sample size and the read dates with any rate, since a small sample could mislead.
Publishing the sample size and read dates with any rate is the right call. Even a small scheduled sample lets you state something concrete, like zero swaps in N re-reads between two dates, which beats saying nothing.
Registry freshness is a security and reliability property, not housekeeping. An entry should carry its publisher, last verified timestamp, supported protocol version, and a clear expiry policy. Clients can then warn or fail closed on stale integrations instead of silently treating a directory as ground truth.
Agreed, and it's measurable. Today a registry entry carries status, statusChangedAt, statusMessage, publishedAt and updatedAt, but no expiry and no "last verified" time (I checked the live OpenAPI spec). So a client can't tell "active, checked yesterday" from "active, nobody has looked in a year". That's why every fact on our pages carries the date we read it.
Freshness needs an owner and a failure mode. Expose last verified time, upstream deprecation status, and a TTL that blocks automatic use after expiry; a stale directory entry should degrade to review, not remain silently selectable.
Agreed on the failure mode. A directory like ours can't block a client, so that has to live in the client or the registry. Today we expose the date we last read each fact, and the status exactly as the registry reports it (we reconcile it against the registry every day). The registry schema itself has no last-verified or expiry field.
One thing we can add on our side, and are considering: a machine-readable "last verified" and "stale after" in our API, so a client can degrade a stale entry to review without scraping pages. I'd only ship it once the refresh behind it runs on a schedule, so the number is one we can keep.
A job that reports ok while the page stays wrong is a nasty kind of bug.
Exactly, and the nasty part is that every check we had was green: the job exited ok, and a version comparison found 0 mismatches across all 51,834 records. The bug lived in a field the check never compared (status, which the registry lets a publisher change without a new version).
Two things changed for us: we now compare every field the source owns, not just the version, and "ok" no longer comes from the exit code. After each run we sample changed servers and read the public page's own marker (version, status, a hash of the status message). It only warns and doesn't roll back, and when our own lookup fails we log "not checked" rather than pass it. The first full compare found 53 stale pages; the next one found 0.
The 'not checked' rather than pass distinction is the part most teams skip — a failed lookup that reads green is worse than a loud failure. When the public page itself is down, does the sample count that as not checked too, or does it warn?
The not checked label is the useful part. A failed lookup should not pass as a clean run.
Agreed. A failed lookup counted as a pass would have hidden the same bug again, so "not checked" is its own state and never counted as clean.
Treating 'not checked' as its own state is the fix. Counting it as clean is how the same bug hides twice.