In my last post, I wrote that when I launched sites on brand new domains, ChatGPT's bots showed up before Google's.
https://hisashi.space/blog/crawler-google-vs-chatgpt/
About an hour after I registered the domain, before there was a single article or a single link, OpenAI's bots were already there.
So how did OpenAI find out that a site had just been born?
My guess was the SSL certificate.
When you get a certificate, its hostname is written to a public log.
I suspected OpenAI watches that log and comes to look.
To test that, I went through the logs of test environments that are not linked from anywhere and have never been submitted to any search engine.
Unlike a production site, a test environment is a place nobody should know about except the people who built it.
What I looked at
One server runs 48 hosts for us. Ten of them are test hosts.
They are test environments for our own products and for client projects we build, and none of them are linked from anywhere.
I am keeping the product names and domains private and calling them Test env A to J.
The data is the server's access log (nginx).
The oldest records go back to the end of April 2026.
All times are Japan time.
You cannot identify a bot by the name it gives (its User-Agent).
These logs alone had 3,305 requests from fakes borrowing the names of OpenAI or Google bots.
So I matched every request against the IP ranges each company publishes, and counted only the ones that matched as real.
・OpenAI: OAI-SearchBot, GPTBot, ChatGPT-User
・Google: Googlebot
・Microsoft: bingbot
Which bots came to the test environments
Real OpenAI bots visited 8 of the 10 test environments.
2,403 requests in total.
Real Googlebot did not visit any of the 10. Not once.
Real bingbot: zero as well.
・Test env A: OpenAI 633, Google 0 (logs from May 10)
・Test env B: OpenAI 519, Google 0 (logs from May 10)
・Test env C: OpenAI 413, Google 0 (created July 6)
・Test env D: OpenAI 396, Google 0 (logs from May 10)
・Test env E: OpenAI 238, Google 0 (created August 20)
・Test env F: OpenAI 175, Google 0 (logs from May 10)
・Test env G: OpenAI 28, Google 0 (created April 30)
・Test env H: OpenAI 1, Google 0 (created October 14)
・Test env I: OpenAI 0, Google 0 (logs from October 5)
・Test env J: OpenAI 0, Google 0 (created April 30)
Googlebot visits our production sites every day.
The test environments under the same domains, on the same server, have not seen it once in more than five months.
What happened on the day they were created
For Test env C, E and H, the logs go all the way back to the day they were created.
In all three, the bots arrived in the same order.
Here is Test env E.
・17:54:00: the first request in the log. Within a few seconds, one scanner bot after another hits the top page
・18:32:44 (38 minutes later): OpenAI's OAI-SearchBot reads robots.txt
・18:32:45: OpenAI's GPTBot reads the top page, then follows its links and starts reading the pages inside
Test env H was the same: OAI-SearchBot arrived 38 minutes after the first request.
For Test env C, it was 2 hours and 12 minutes.
On Test env C and H, one of the bots in those first few seconds called itself CT-WP-Scanner.
CT stands for Certificate Transparency, the public log of SSL certificates.
When a certificate is issued, its hostname goes into this log, and anyone can read it.
The hostnames of all three test environments were in that log.
At that point, the only places anyone could learn those hostnames were the certificate log and the DNS records.
No links, no sitemap, no search engine submission.
Google and OpenAI behave differently
Google explains how Search works starting from discovering pages through links and sitemaps.
https://developers.google.com/search/docs/fundamentals/how-search-works
The test environments have no links pointing to them and no sitemap submitted anywhere.
And in the logs, Googlebot never came.
OpenAI's bots came to places we had told nobody about, within an hour of creating them.
On Test env C and E, GPTBot started reading the pages one second after OAI-SearchBot read robots.txt.
My take
From here on, this is my interpretation of the logs.
OpenAI treats a new domain or a new host getting an SSL certificate as a signal, and comes to look.
Googlebot behaves differently.
In other words, I think OpenAI collects information more widely than Google does, and analyzes it on its own.
OpenAI has not said how it finds new sites.
What the logs show is this: once a hostname appeared in the certificate log and got its first request, OpenAI's bots arrived within 38 minutes to about 2 hours. And Google has not come to similar places in more than five months.
Limits of this data
・I cannot fully rule out that one of these hostnames was written somewhere I do not know about
・For the test environments whose logs start on May 10, I cannot see what came before that
・Some places, like Test env J, did not get OpenAI either
・There is one more test environment whose only surviving logs are from April 2025. Googlebot came 54 times and bingbot 20 times. Since the period is different from the others, I left it out of the ten
How to check your own site
You can count them in your server's access log like this.
grep "OAI-SearchBot" access.log | wc -l
grep "GPTBot" access.log | wc -l
grep "Googlebot" access.log | wc -l
Any program can use any name.
To know whether a bot is real, match it against the IP ranges each company publishes.
・OpenAI: https://openai.com/searchbot.json , https://openai.com/gptbot.json
・Google: https://developers.google.com/static/search/apis/ipranges/googlebot.json
This is how you can see whether OpenAI's bots are visiting your test or staging environments.
Wrapping up
Out of 10 test environments that are not linked from anywhere, OpenAI's bots visited 8.
Googlebot visited none.
On the days they were created, OpenAI's bots arrived between 38 minutes and about 2 hours after the first request.
Even an environment you think is private gets found by OpenAI the moment you get a certificate for it.
Top comments (0)