I Tested 5 AI Engines On My Own Sites. None Agreed.
In July I wrote that my open-source LLM visibility checker tested Claude only and th...
For further actions, you may consider blocking this person and/or reporting abuse
the
web_search=truegotcha is the one that trips every team doing multi engine visibility tracking. ChatGPT with web search off returns a fluent answer with zero citations. reads as 0/10 when it's really "tool not configured." made the score meaningless until we caught the config path.the Gemini 2/10 while everything else is 0 is the interesting result. Gemini's live index plus training data blend means content that hasn't built enough citation mass for ChatGPT retrieval can still surface there first.
does your tool normalize for query type? "hire freelance X developer" queries pull marketplaces not personal sites — curious if branded vs intent queries show a different signal pattern.
Mudassir, checked the code before answering: zero query-type normalization. Every query just runs through all five engines the same way, no classification step.
That's a real gap. Undermines the dannwaneri.com numbers specifically: most of those queries were "hire freelance [library] developer" stuff, marketplace intent, not brand intent. Upwork and Fiverr are the expected answer there, not a personal site. naija-vpn.com's queries skewed more informational, brand-adjacent, which is probably part of why that domain scored anything at all.
Your live-index-plus-training-blend theory for Gemini's interesting, but I can't verify the retrieval architecture, so I'd hold that as a hypothesis, not fact. It does line up with the one near-miss I found though: Gemini citing my /hire-python-developer/ page for a scipy query. Reads more like live-retrieval adjacency than something trained in.
What would you use as the marker for transactional versus brand intent, if you were building the normalization step??
for classification at the query level: search modifier presence is the cleanest signal. "hire", "services", "near me" → transactional. "what is", "how to", "vs" → informational.
for GEO/AEO specifically i’d also check whether the query names a proper noun. brand named queries behave differently from intent queries even when both look informational.
the harder bucket is “branded informational” — that’s where personal site benchmarks actually live and standard classifiers miss it.
are you open sourcing the normalization step when you build it?
dannwaneri.com/robots.txt and /llms.txt both return the homepage SPA HTML (~36k, title "Remote AI Agent Developer") — not text files. That is why Claude/ChatGPT can score you 0/10 while Gemini still cites /hire-python-developer/. On naija-vpn.com, robots.txt is a real Allow: / but /llms.txt also falls through to the homepage, and the brand is NaijaVPN while the title is "Get Paid Internationally in Nigeria."
If you want I can rerun the same three engines on both domains after you ship real text files.
RileyCraig14 thanks, checked it myself: dannwaneri.com/robots.txt and /llms.txt both return the same 35KB SPA shell, title and all. naija-vpn.com/robots.txt is a real 67-byte file, /llms.txt falls through the same way. Matches exactly.
Root cause is almost certainly the build - those files never made it into the deployed output, so the SPA catch-all swallows the request. No excuse for it being broken this long.
Might be a better explanation than training-data cutoff for why Claude scored 0/10 on both domains in the piece. If it's checking llms.txt as a discovery signal and getting garbage HTML back, that's a harder failure than "hasn't seen this yet."
Shipped real files for both - dannwaneri.com/robots.txt and /llms.txt, naija-vpn.com/llms.txt. Taking you up on the re-run. Did Perplexity or Copilot behave any differently around llms.txt, or was this a Claude/ChatGPT/Gemini story?
The multi-engine result is the useful part here. AI visibility is starting to look less like one ranking problem and more like measuring five noisy distribution channels with different retrieval habits. I would be curious whether the disagreement is stable over time, or whether the same domain flips month to month as each engine changes its sources.
Dean, same open question as Bulti raised above, just aimed at time instead of repeat-runs. I only have a single snapshot, no month-over-month data yet. The "noisy distribution channel" framing implies real drift as each engine's index and retrieval change independently of your content - worth testing directly, not assuming.
Would you expect Google's own AI Overview to drift less than these five, since it isn't routing through a separate retrieval layer at all?
The zero overlap is useful, but with 10 queries the bigger design risk is conflating engine coverage with run stability. In a 284-brand Korean DTC scan we ran across 50 AI shopping questions per brand, 65.5% of brands had zero appearances and the mean was only 0.648 out of 50. That sparse distribution made the denominator and repeatability as important as the engine split. For Copilot, I’d report 0/9 answered plus 1/10 unavailable rather than only 0/9, then rerun the identical prompts two or three times. Have you checked whether the Gemini/ChatGPT ownership flip persists across repeated runs?
Bulti, no. Single run per query, no repeats. Didn't check whether the split holds.
10 queries once each tells you almost nothing about whether Gemini citing dannwaneri.com is stable or a coin flip that happened to land there that day. Your 0.648-of-50 mean is a good reminder of how sparse this signal actually is at scale. At n=10 with one pass, I'm probably closer to noise than to a measurement.
The 0/9-plus-1-unavailable framing is right too. Folding the Copilot error into the denominator hides that it's a different failure mode than "answered, didn't cite you."
At 284 brands times 50 questions times multiple runs, what's your actual query volume before rate limits become the bottleneck?
Permit me to say that we will keep building imperfect systems. People will keep discovering the cracks, and some will genuinely enjoy fixing them.
Those little cracks are already systemic because the architecture doesn't distinguish statistical matching from deterministic verification and provenance.
that's exactly what the Gemini scipy-to-python near-miss in the piece was: statistical adjacency standing in for actual verification.
So do you think this is a problem we can actually solve, or is it something we can only ever barely manage?
I don't know for now. Ask me after a few hundred more runs.
The disagreement between engines is the part worth mining, since it maps exactly where each retriever and ranker splits on the same query. When I ran a similar test the spread only made sense after I logged which source each engine cited, and then it went from spooky to debuggable. Did the gap widen more on ambiguous queries or on the ones with a clear answer?
Kartik, honestly, most of my queries were narrow, not ambiguous. "Hire freelance scipy developer" has one obvious answer domain. So no real ambiguous-vs-clear split to test your hypothesis against yet.
The one case that came close: Gemini cited my /hire-python-developer/ page for a scipy query. Not ambiguous, but adjacent - a retriever generalizing across a topic boundary instead of matching exactly. That's the shape your framing predicts.
I do have the cited source per engine per query logged already, just haven't sliced it by ambiguity. What counted as ambiguous in your test - topic breadth, or genuinely competing correct answers??
that is why worlds need something which can prove the ai's (ai agents, ai models) responses,