What a crawler can retrieve from nine small product sites: robots, sitemaps, JSON-LD and llms.txt

This article was created with the help of AI.

When you run several small product sites, it is easy to assume that robots rules, sitemaps, structured data and language annotations are all in place. We checked instead, once, on all nine public sites of the IVRYN portfolio, and published the result as a dated snapshot.

What was collected

One automated pass, without authentication, fetched each public homepage, robots.txt, the root sitemap, the blog sitemap and llms.txt. It parsed the XML locations, the JSON-LD types, the hreflang values and the root-path crawler rules. The observation is dated 2026-08-14 (00:22 UTC).

Observed results

In short: 9 of 9 homepages returned HTTP 200, 9 of 9 roots allowed OAI-SearchBot, and the sitemaps declared 760 unique URLs in total.

Versioning

The dated directory is immutable by convention. CSV and JSON files come with checksums, so a later snapshot can be compared line by line. The current URL may be refreshed by a future observed snapshot.

What this does not tell you

This is a complete census of one owner-operated portfolio, not a representative sample of websites, startups or production stacks. It describes retrievable public artefacts at one point in time. It does not measure search placement, backlink authority, audience, adoption, revenue or the probability of citation by a search or AI system.

A crawler being allowed does not mean it visited, indexed, ranked or cited a page. A sitemap declaration is not proof of indexation.

Data and method

The CSV, JSON, field dictionary and reproduction notes are on the original page: Nine-site public web transparency snapshot. Protocol references: RFC 9309 (Robots Exclusion Protocol), the Sitemaps XML protocol and Schema.org Dataset.

Do you run a similar check on your own sites, and which field would you add?