The finding
There is a story going round that brands are invisible to AI assistants because their crawlers are being blocked. I went looking for it. On this sample it is not true — and the real problem is duller and more fixable.
0of 38 measurable sites refused
OAI-SearchBot,
ChatGPT-User, PerplexityBot or ClaudeBot while their own
robots.txt allowed it34%served no structured data at all — nothing on the page
states a fact in a form a machine does not have to infer
Two sites refuse Bingbot at the edge while inviting it in robots.txt, which also means Copilot. That is the whole of the access problem here.
Quotability is where the sample actually fails. Thirteen of the thirty-eight published zero
JSON-LD; fourteen had no Organization markup; nine had no <h1> at
all. Two had FAQ markup. Two, out of thirty-eight.
That matters because the two dominant systems work differently. Perplexity retrieves live and rewards pages with clear structure. ChatGPT leans on what it already absorbed — brand mentions across the web correlate at roughly 0.66 with citation rates. Neither is helped by a page that says nothing machine-legible about who you are.
Two corrections to this page, both mine
This report has been wrong twice in one day. Both fixes are recorded here rather than quietly applied, because a measurement report that hides its own corrections is not worth reading.
The control condition, caught before publishing
My first pass counted a site as blocked whenever robots.txt allowed a crawler and the live request failed — without requiring the plain-browser control to succeed. That produced “30% of sites block AI crawlers”. It was wrong by a factor of three: plenty of sites refuse any datacentre address regardless of who is asking, and refusing my machine is not the same as refusing OpenAI's.
A misidentified company, caught two hours after publishing
The first version of this page named pleo.com as its textbook case: a browser got
209,642 bytes, five crawlers got 403. The measurement was real. The identification was not.
pleo.com belongs to P. LEO Group, an electrical-insulation manufacturer founded
in Hong Kong in 1973 — not to Pleo, the expense-management company, which is at
pleo.io and which serves 200 to all eight crawlers. It was in my list by
mistake and it was never a member of the population this page describes.
It has been removed, pleo.io measured in its place at 11:48 UTC, and
every figure on this page recomputed from the raw data. Doing that changed the headline:
the count of sites refusing the answer-engine crawlers went from one to zero.
I am leaving this here in full because it is the more useful of the two mistakes. A domain that looks like a brand is not that brand, and a report that names companies has to check which company it is naming before it publishes. Mine did not, for about two hours.
Method
The method is the product, so here it is in full.
For each domain I fetched the home page once as an ordinary desktop browser — the
control — then once as each of six crawler user-agents, from the same machine, within the same
minute. I fetched /robots.txt and evaluated it per agent using longest-match
Allow/Disallow with the * fallback, recording the exact rule behind each verdict. I
fetched /llms.txt and checked whether a 200 response was genuinely plain text rather
than an HTML error page wearing a success code. Then I parsed the served HTML for title, meta
description, heading structure and every JSON-LD block, counting the ones that fail to parse.
What counts as a finding
robots.txt allows the agent, and the server returns 4xx or 5xx to it, and the control returned 200.
That third condition is the whole discipline, and it is what the first correction above was about. Without it the figure comes out three times too large.
Sampling, stated plainly: fifty recognisable direct-to-consumer and SaaS brands, chosen by hand so a reader can judge each result against a site they know. This is a convenience sample, not a random one. It does not represent the web, and the figures below describe these sites and nothing else.
What I measured
36 / 38robots.txt declares a sitemap
26 / 38llms.txt present and genuinely plain text
14 / 38no Organization structured data
13 / 38no JSON-LD of any kind
9 / 38no <h1> on the home page
5 / 38more than one <h1>
2 / 38no meta description
2 / 38refused a crawler robots.txt allowed
2 / 38under 1,000 bytes of text without JavaScript
2 / 38FAQ structured data present
0 / 38JSON-LD that fails to parse
The llms.txt result surprised me. More than two thirds of these brands publish one, which is faster adoption than a proposed convention with no known consumer usually gets. Sizes run from 908 bytes to 377,217 — one of them is not a summary, it is a second website.
The two that refuse a crawler they invited
Named, because the whole point is that you can check it.
notion.so and n8n.io
A browser gets a full page from both — 241,360 and 342,480 bytes. Bingbot
gets 403 from both, from the same address, seconds apart, with no rule against it in either
robots.txt. That also means Copilot, which runs on Bing's index. Nothing in either site's
own configuration says this is happening; something in front of the site is doing it silently.
This is the failure worth knowing about: it is invisible from inside the site. A robots.txt checker gives it a clean bill of health.
figma.com and loom.com — a different column
figma.com disallows ClaudeBot and GPTBot in robots.txt; loom.com
disallows GPTBot. Those are deliberate, visible choices, correctly configured and
working as intended. They do not belong with the two above and are not counted with them.
The twelve I could not measure — and why that is the interesting part
Twelve of the fifty refused my machine outright. Not the crawlers: everything, including the plain browser control — 403, 429 or 503 on the first request.
bombas · ridge · away · warbyparker · lush · patagonia
· therealreal · hims · figs · tushy · olipop ·
personio
I cannot tell you whether those sites block AI crawlers. Neither can any other remote scanner, including every “AI visibility checker” that runs from someone else's servers. They will report a result anyway. It will be about their own IP reputation, not about your site.
This is the one structural argument for checking from inside: a tool running on your own server, requesting your own pages, is never mistaken for a scraper. Nearly a quarter of this sample is invisible to the alternative — which is why I wrote a plugin that runs on your site instead of mine.
What I would tell an operator
Check access once, properly, then stop worrying about it. On these numbers you are almost certainly fine. If you are not, it is a setting in front of your site rather than anything in your code, and it takes minutes.
Then make the page quotable. One <h1> that names the subject.
Organization markup that says who you are. Answers written as answers, with the
question above them. A third of these brands have none of that.
Do not believe anyone who promises you a recommendation. What gets a brand named by ChatGPT is mentions across the web it already trusts — earned, not installed. Anything you put on your own server makes you easier to read and easier to quote. That is real, and it is not the same thing.
Measured 2 September 2026 between 11:00 and 11:20 UTC from a single host, with pleo.io measured at 11:48 UTC when it replaced the entry removed above. Every figure is a measurement of what those servers returned that morning; sites change, and a re-run tomorrow may differ.