We make about seven requests to your site, read nothing but the homepage, and store none of it. Every result below is a live measurement — where something could not be measured, we say so rather than guess.
robots.txt Is a Request.
The Fetch Is the Proof.
1. We parse robots.txt properly.
Not with a keyword search — to the actual standard (RFC 9309).
Consecutive User-agent: lines are grouped so their
shared rules apply to every agent in the group; the most specific
group wins; within it the longest matching path wins, and on an
exact tie Allow beats Disallow. An empty
Disallow: means allow everything. We tell you which
group matched and which line decided it, so you can check our
working. We do that for twelve named agents.
2. Then we actually fetch your site as each bot. This is the part that matters, because robots.txt has no enforcement power at all — a crawler that ignores it still gets your page, and a crawler that respects it still gets turned away if your firewall says no. We send each vendor's real published User-Agent and report the true status code. A 403 here is the big one: that is a WAF rule or a bot-management toggle refusing the crawler at the edge, and it will not appear anywhere in your robots.txt.
3. We run a control, so the answer means something. We make the same request one more time with an ordinary browser User-Agent. If the browser gets 200 and GPTBot gets 403, only the User-Agent changed — that is selective blocking of AI crawlers, proven. If everything fails, we say the result is inconclusive rather than blaming AI blocking for a site that is simply down or refusing datacentre traffic. Without the control a 403 could mean anything, and we would rather report nothing than report something we cannot stand behind.
4. Then the supporting checks.
Whether the page is readable without JavaScript, whether your
business is identified in structured data, whether an
llms.txt exists — and whether a stray
noindex is telling the world to ignore you. We found
exactly that on a live law firm's site once: a dev-mirror config
with Disallow: / had shipped to production and had
been telling Google to go away for months. Nobody had looked,
because nothing on the page looks broken.
More than half the live business sites we have tested were quietly turning AI crawlers away — nearly always at the firewall, and nearly always without the owner knowing. We wrote up how that happens.
What These AI Crawlers Are,
and When Blocking One Makes Sense.
If you have just found one of these names in your server logs, here is the plain-English version — what each one does, and whether turning it away is a reasonable call or an own goal. The short rule: bots that put you into an answer are worth allowing; bots that only take content for model training are a business decision, and a legitimate one either way.
ClaudeBot
Anthropic
Anthropic's crawler, and by a wide margin the AI bot people
search for most after finding it in their logs. ClaudeBot
collects web content that may be used to train Claude.
Anthropic also runs Claude-User, which fetches a
page because a person asked Claude about it right then, and
Claude-SearchBot for search indexing.
Blocking it: defensible for ClaudeBot if you do
not want your content in training data. Blocking
Claude-User is different — that one is a real
person trying to read your page through Claude, and blocking it
costs you a visitor.
GPTBot
OpenAIOpenAI's training crawler. Disallowing GPTBot tells OpenAI your content should not be used to train its foundation models. It is the single most commonly blocked AI user-agent on the web, and it is often blocked by accident — a managed "block AI bots" switch at the CDN catches it along with everything else.
Blocking it: reasonable if training use is your concern. Just be clear that it is a separate decision from appearing in ChatGPT's answers — that is the next one.
OAI-SearchBot
OpenAIThis is the one that decides whether you show up when someone asks ChatGPT a question. It is entirely separate from GPTBot: you can allow search and disallow training, and OpenAI documents that combination explicitly. Sites opted out of OAI-SearchBot do not appear in ChatGPT search answers.
Blocking it: almost never a good idea. This is pure visibility — the AI equivalent of blocking Googlebot.
PerplexityBot
PerplexityPerplexity's search crawler. It exists to surface and link websites in Perplexity's answers, and per Perplexity's own documentation it is not used to collect content for training foundation models. Perplexity cites its sources prominently, so being indexed here tends to send real clicks.
Blocking it: hard to justify. You lose the citation and the traffic that comes with it.
Google-Extended
GoogleNot a crawler at all — a robots.txt control token. Google already has your pages via Googlebot; Google-Extended only says whether that content may be used to train and ground Gemini. Disallowing it has no effect on your Google Search ranking, which is the detail most people get wrong.
Blocking it: a clean, no-cost opt-out of AI training that leaves Search untouched. Plenty of publishers do exactly this.
CCBot
Common CrawlCommon Crawl is a non-profit that publishes a free, open archive of the web. It does not build a product from your content — but nearly every AI lab has trained on its archive at some point, so in practice CCBot is an upstream source for many models at once.
Blocking it: a broad, blunt way to stay out of many training sets. It also removes you from an open research archive, which some site owners value being in.
Access Is Step One.
Being Worth Quoting Is the Job.
Getting the crawlers in is the part you can fix this afternoon. Getting named in the answer takes entity schema, content an AI can actually cite, and measurement that tells you whether any of it worked. We do the whole job — and we test everything instead of assuming, which is the same reason this tool runs a control request.
While you are here: our free llms.txt generator builds a valid llms.txt from your real pages, and the GEO guide covers what actually moves the needle in AI search.