In July I set out to add our MCP server to the ChatGPT app directory. The server was already live, running in Claude and Cursor, with OAuth working...
For further actions, you may consider blocking this person and/or reporting abuse
The sandbox target validating writes without committing side effects is the smartest thing on this list. When an MCP server exposes state-mutating tools, giving reviewers a mock sink that still runs the real parameter validation avoids needing disposable test accounts, while also doubling as an integration test in CI.The multi-catalog drift across Zed, Docker, and ChatGPT manifests is the other headache that bites hard. Once tool schemas are maintained manually in more than one place, a renamed parameter or dropped tool inevitably rots in some README. I ended up writing a quick CI check that diffs catalog manifests against the server's live reflection output, which catches mismatched tool lists before anyone opens a submission PR.
Two people in this thread landed on the same CI check independently — diffing the catalog manifests against the server's live reflection — which is a pretty strong sign that's the actual fix, not just remembering to update more READMEs.
And you're right that the playground doubles as an integration test — I'd only been thinking of it as a reviewer convenience. Same mock sink, run in CI, and every state-mutating tool gets its real validation exercised on every push without touching a real account. Adding both. Good thread to have posted this into.
The
tools/listwithout-auth requirement is the sharpest client divergence to know about: Claude and Cursor both authenticate before requesting the tool list, so discovery-behind-a-token works fine there and fails silently elsewhere. The safest default is treatingtools/listas a public menu and auth as a gate on execution, not on discovery. The annotation exhaustiveness issue - scanner requiringdestructiveHint: falseeven when the spec allows omitting it on read-only tools - suggests ChatGPT's submission pipeline is running stricter validation than the protocol itself mandates, which is worth anticipating early if you're targeting any other app directory with a similar scanner step."Discovery is a public menu, auth gates execution" is the cleanest way to say it — I'm stealing that framing. It's obvious in hindsight, but the spec lets you gate discovery too, so nothing tells you it's a bad idea until a client that reads the menu first walks away with an empty plate.
And yeah — the scanner being stricter than the protocol is the thing I'd warn anyone about before their next directory. Passing the spec isn't the bar; passing someone's stricter reading of it is. I'm now assuming every catalog has its own extra layer and building the annotations to the strictest one I've hit, rather than the one the spec technically allows.
The drift half looks like it is still open in the artifact the other descriptions were derived from.
src/index.jsonmainhas 18server.toolregistrations, and the four you named are all still among them —linkedin_post_stats,linkedin_account_stats,linkedin_followers,linkedin_profile_summary— with the last push dated 2026-08-04, before the rejection.docs/tools/overview.mdheads that group "LinkedIn Analytics (6 tools)", counting the two reaction tools with them, so it is a third independently written count sitting in the same repo. That matters more than a stale README because the repo is what the Zed extension and the catalog PR were copied from, so the next person describing the server re-derives the original claim rather than reading your corrected release notes. The useful part is that your first fix makes this one mechanical: oncetools/listhas to answer 200 with no token, the live list becomes machine-readable authority, so a CI step diffing the registered names against it turns "we described tools an MCP user cannot reach" from a review finding into a failing build.Good catch — you're right that the repo itself still has all 18, so I fixed the descriptions and not the thing they're copied from. The "Analytics (6 tools)" heading is a third count in one repo, which I'd missed entirely.
The CI idea is the real fix though: once tools/list answers 200 with no token, the live list is machine-readable, so a build step diffing the registered names against it turns this into a failing build instead of a review finding. Adding that. Thanks — genuinely useful.
The reviewer-account thing is the part people miss most. You can pass every spec check and still fail because you tested as yourself, not as a stranger with no context. That's basically the whole trust problem with MCP servers in miniature: works great when the person running it already knows how it's supposed to behave, breaks when someone with zero prior context tries to use it. We're seeing the same pattern building a marketplace for these servers, the technical spec compliance is the easy 80%, the "can an outsider actually operate this safely" part is where everything interesting (and broken) lives.
Yeah — "works when you already know how it should behave" is exactly the trap, because the person who built it can't un-know how it works. You test it and it passes, because you're the worst possible reviewer: you already know all the parts you'd otherwise get stuck on. A stranger with no context is the only honest test, and they're the one person you can't easily be.
The 80/20 split matches what I saw too. Spec compliance was a checklist I could finish. "Can someone with no idea what Publora is operate this without help" had no checklist, and that's where the whole month went. Curious what the marketplace side looks like — you're seeing this across a lot of servers at once, which is a vantage point I don't have from inside one.
the google sign-in one is the trap that gets everyone. you test your own oauth flow a hundred times and it always works because you're always logged in as you. never occurs to check if a stranger with your exact instructions could actually get in.
Yeah — the cruel part is that testing it more doesn't help, it makes it worse. Every successful run builds confidence in a flow that only works because it's you running it. A hundred green passes and not one of them touched the thing that was actually broken. The only useful test was the one I kept skipping: hand the exact steps to someone who isn't me and watch where they get stuck.
The tools/list-before-auth thing is the one that'll bite the most people. It's such a small ordering difference between clients but it turns "works everywhere" into "works nowhere" for one specific platform. We're building an MCP discovery layer and this is exactly the class of problem that never shows up in the spec, only in the gap between what different clients assume. The reviewer-account point is underrated too. Testing as yourself is the default failure mode for basically every integration, not just MCP.
Yeah, "the gap between what different clients assume" is exactly it — the spec says what's allowed, not what each client actually does with that freedom. Claude and ChatGPT both read the spec correctly and still ended up on opposite sides of one ordering choice. A discovery layer sitting in that gap sounds genuinely useful, because right now everyone rediscovers these differences one rejection at a time.
And yeah — "test as yourself" is such a quiet default. You never notice you're doing it, because for you the thing works. Curious what you're building, MCP discovery is a good place to be right now.
The "tools/list before auth" quirk is the one that'll keep biting people, because it's not a client bug, it's a real disagreement about what tools/list means. ChatGPT treats it as "what could this server ever offer," Claude treats it as "what can this authenticated session offer." Both are defensible reads of an unauthenticated endpoint. We hit a version of this building an MCP layer across 1,500+ third-party apps: some upstream APIs literally can't tell you what's callable until you're authed (permission-scoped endpoints), so an unauthenticated tools/list is either a lie (static list, may not match reality) or empty (which ChatGPT then can't recover from per your post). There's no spec answer to that yet, only client-specific workarounds. Curious whether OpenAI's scanner team has said anything about fixing the empty-list dead-end rather than pushing servers to fake a static list.
You've put your finger on the thing my post quietly assumed away. My advice — "make tools/list answer 200 without a token" — only works because our tool list is static. For a permission-scoped server it falls apart exactly the way you describe: the unauthenticated list is either a lie or empty, and ChatGPT can't recover from empty. So "just expose it publicly" isn't a fix there, it's a choice between two wrong answers.
"What could this server ever offer" vs "what can this session offer" is the cleanest framing of the disagreement I've seen. And the spec genuinely doesn't resolve it — it says the endpoint can require auth, without saying what a client should do when it does.
No, I haven't seen anything from OpenAI's scanner team about the empty-list dead-end — and honestly that's the part worth pushing on more than the static-list workaround. Recovering from an empty discovery is a client problem they can fix once; asking every permission-scoped server to fake a static list is a workaround they're pushing onto everyone forever. If you ever get a channel to them on this, I'd love to know what they say.
Great work!
Thanks!
The tools/list ordering difference is a great example of why protocol compliance and real client interoperability need separate preflight tests
Exactly — passing the spec tells you the server is correct, not that any given client will accept it. Those turned out to be two different green checkmarks, and I only had a test for the first one. A preflight that actually runs the real client's discovery order would've caught this before submission instead of after a rejection.