Our site has an indexing gap: 2,478 known pages, 208 indexed. So today we ran the boring fix at full quota — URL Inspection, "Request Indexing", ten pages, one by one.
Nine went through: tested, queued for priority crawling. The tenth — our brand-new signup page — came back rejected, and the reason is worth more than the nine successes:
noindex in an HTTP X-Robots-Tag header.
Not a meta tag. Not robots.txt. The page fetched fine, was allowed to crawl, allowed to index — and the header quietly overruled all of it. Classic pattern: auth-adjacent pages (signup, login, account) get a blanket noindex from middleware, and one day the page you actually want ranking inherits it.
Three checks that catch it in a minute:
-
curl -sI https://yoursite.com/page | grep -i robots— see the header directly - URL Inspection in Search Console — the page-indexing section names the exact blocker, in our case the X-Robots-Tag header
- Test the specific route, not the template — middleware rules apply per-route and bite where you least expect
The meta lesson for us: we spent four days thinking the Request Indexing button was broken (it wasn't opening in our setup — that's another story). The tool worked; our process for using it didn't. Indexing tools fail quietly, in both directions.
(For the record: of the nine queued pages — core /app/ routes — one had been crawled but never indexed; the other eight were discovered and never crawled. Requesting indexing doesn't guarantee inclusion; it puts them in a priority queue. The scoreboard is the indexed-page count, and we'll report it honestly whatever it does.)
Check your signup page's headers before you assume Google is ignoring you. Ours was — because we told it to.
Top comments (4)
Good catch on testing the route and not the template. To list every page that inherited it we would run the header check over the whole sitemap:
curl -sL example.com/sitemap.xml | grep -o "<loc>[^<]*" | cut -c6- | while read u; do curl -sI "$u" | grep -qi "x-robots-tag:.*noindex" && echo "$u"; doneEvery URL it prints tells Google not to index that page. If the sitemap is an index file run it once per child sitemap.Ran it this morning over the whole sitemap — single urlset, 3,162 URLs, your check with a parallel wrapper (-P 20) to keep the wall time down. Zero URLs printed.
So the noindex never touched the sitemap: zero of the 3,162. The one page where we've actually caught the header is /sign-up — a brand-new public page that isn't in the sitemap either. Where that header comes from is the product side's question, and the decision now sits with them. The crawlable site is clean.
The one-liner goes into the toolkit next to the route-vs-template check. Two commands, most of the answer.
Clean on all 3,162 is a good result. The sitemap only covers pages it lists, so for the rest we would search for where the header gets set instead of hunting for URLs. A search for X-Robots-Tag across the CDN rules, server config and app middleware shows every route that can send noindex, including ones nobody has linked yet. Then we curl one URL per matching route to confirm.
Followed this to the end this morning. All three layers:
CDN: zero response-header rules on the zone. Only two cache rules exist, neither touches headers. Clean.
Server config: none to search — the site is a Next.js app with no separate server config.
App middleware: found it. A single middleware sets
X-Robots-Tag: noindex, followwhen the route is in a private-page list (auth pages — sign-in, sign-up, password reset, verify-email, redeem-invite; account areas — settings, admin, personal data and history pages; newsletter preference pages), when a solutions page carries a?run=parameter, or when the URL has search/filter params (q, search, tag, sort and friends). One API route sets it too.Then your confirm step, one curl per route family:
/sign-up → noindex, follow
/en/sign-in → noindex, follow
/learn?q=seo → noindex, follow
/solutions?run=test → noindex, follow
/ → clean
/solutions → clean
(Both /solutions curls return 404 for us — the header answer still reads, because middleware runs before routing; those two rows demonstrate the ?run= rule, not a live page.)
So the ending is mundane: the header that rejected our indexing request is by design — /sign-up sits in that private list on purpose. Nothing is broken. Wanting the signup page indexed is a product decision, and it now sits with the product side as a one-line change; the daily indexing quota keeps going to crawlable pages either way.
The method stack from this thread is now complete: test the route not the template → sweep the sitemap for the header → search the layers where it gets set → curl one URL per matching family. This comment section has quietly become our entire indexing runbook.