I pre-registered a study on AI visibility signals. The main result was null.

Originally published on angeo.dev. Full tables, p-values and the sealed plan are there.

Most claims about AI visibility are untestable by design: publish the signals, wait, attribute anything good that happens to the signals.

I wanted a version I could not fudge, so I wrote the analysis plan first, hashed it, and sent the hash to the other party before I had any data.

The question

Do businesses AI assistants name repeatedly differ, on observable technical signals, from businesses the same assistants name once?

Every business in the corpus was named at least once, so this says nothing about how to enter an answer. It compares repeat against one-off mentions inside a named-business corpus.

Four signals, all externally observable:

Study setup

The answers came from a partner (connexion.me), who ran 44 product-level home-decor buying questions across ChatGPT, Gemini and Perplexity, twice, in two arms — 264 answers per arm.

Blinding was deliberate. I did not write the questions and did not see their store list until my plan was sealed; they never saw my frame, my scan results or my thresholds.

Roster rows                                    669
  no resolvable domain                        -186
  resolved to a different company               -3
  marketplaces and listing surfaces            -12
  duplicate rows collapsed                     -10
Unique domains analysed                        458
  scanned successfully                         455

Cases: 3+ mentions across both runs and present in both. Controls: exactly one mention across both runs. Head excluded first — anything in 53+ of 264 answers (Amazon, Etsy, Wayfair, Target, Home Depot).

The pre-registration

Sealed 10 August, SHA-256 9b4ccf12629e…:

• Under 15% of named businesses would be Magento
• No signal would separate the groups by more than 15 points
• Refutation condition: any signal differing by 20+ points with the named group higher

Result — generic arm

Two-sided Fisher exact. Percentages are of stores where the signal was observable.

The refutation condition was not met — in this arm, in the context arm, or pooled. No signal separates the repeatedly named group from the once-named group in the direction the field assumes.

Magento share of the 455 scanned domains: 13 stores, 2.9%.

The one that went backwards

llms.txt is more common among businesses named once than among those named repeatedly. Same direction in all five cuts, diffs of −15.3 to −17.4, nominal p between .012 and .030. Four signals across five cuts, no multiplicity correction — report them, do not treat them as confirmatory.

Before reading that as "llms.txt hurts you", I fetched the flagged files and read the first line of each:

113 of 159 llms.txt files — 71% — matched the same generated heading, differing only in the brand name. Not 113 independent decisions to publish. One generator.

Narrowed to the analysis universe: 146 of 455 serve the file (32%); 104 of those 146 are the template. Strip it and adoption is 9%, against 11% on my separately measured frame of 762 Magento stores.

I was about to publish 32% as adoption. Most of the apparent difference between the two populations was boilerplate.

The engineering lesson, if you want one

Three things I would do again on any measurement like this:

1. Write the failure condition before the data exists, and publish it when it fires. Mine fired in a secondary cut on a case group of four stores. Uninterpretable — and reported anyway, because a refutation condition you drop when it is inconvenient was never one.
2. Record presence, then inspect content separately. The scanner keeps no page content by policy. That is correct for a crawler, and it is also why the template finding needed a second, explicit pass.
3. Watch what your sampling heuristic does. In a related panel, 12% of stores had a different product page sampled on the second visit. That alone manufactures schema "changes" that never happened.

Caveats that matter

• Two of four signals need a product page. Effective samples: 10 of 55 cases, 46 of 181 controls. Too small to conclude anything, and the missingness is probably not random — discovery works worst on large sites, which dominate the case group.
• Losses are not random: shops behind bot mitigation returned 403/429, and those shops may also be more likely to block AI crawlers.
• Crawler access is one route into an answer, not the only one — indexes, caches, feeds and listing sites all bypass it.
• One category, three engines, two days.

Reproducing it

Sealed plan and both amendments with hashes, the marketplace exclusion list, the dropped-domain list, the per-business analysis rows and the scanner are all published. The scanner identifies itself, obeys robots.txt, keeps no page content and can be blocked in two lines.

Full write-up with every table: AI assistants named 458 shops

Earlier instalments: 462 stores scanned in July, 762 in August · 376 stores measured twice, 94% unchanged

I maintain a set of MIT-licensed Magento 2 modules for this layer — llms.txt generation, robots.txt AI-bot management, JSON-LD, MCP and UCP endpoints — on Packagist under angeo/. If you have run a similar measurement and got a different direction on llms.txt, I would genuinely like to see it.