Michael D. Wandzilak
This year we audited the website of an established business that wanted to show up in ChatGPT’s recommendations. Their robots.txt politely allowed every AI crawler. Their content was good. And every single AI system — OpenAI, Anthropic, Perplexity — was getting a 403 Forbidden at the front door, served by a Cloudflare setting the owner didn’t know existed. ChatGPT couldn’t read a word of the site. If your business runs behind Cloudflare — and millions do — there’s a real chance this is happening to you right now. Here’s how to find out in five minutes.
Why this happens to well-configured sites
Cloudflare offers AI-bot blocking, and at various points it has been enabled by default for new zones and via one-click “Block AI bots” managed rules. It was pitched as protection against content scraping — reasonable for some sites. But it makes no distinction you’d actually want: the same switch that stops a scraper also stops OAI-SearchBot, the crawler that decides whether ChatGPT’s search can cite you, and ChatGPT-User, the fetcher that reads your page live when a user asks about you. Blocking those doesn’t protect your content — it removes you from the answers.
The nasty part is the layering. In the account we audited, the new AI-bot policy panel said Allow — and an older, legacy “Block AI bots” rule scoped to all pages was quietly overriding it. Two dashboards, opposite answers, and the only truth is what the server actually returns. Which is why you test from outside.
The five-minute check
robots.txt is a request; the firewall is a decision. Test what the server does, not what your settings say. From any terminal (Mac: Terminal.app), run these against your own domain — each one pretends to be a specific AI crawler:
curl -s -o /dev/null -w "GPTBot: %{http_code}\n" \
-A "GPTBot/1.0" "https://YOURDOMAIN.com/?cb=$(date +%s)"
curl -s -o /dev/null -w "OAI-SearchBot: %{http_code}\n" \
-A "OAI-SearchBot/1.0" "https://YOURDOMAIN.com/?cb=$(date +%s)"
curl -s -o /dev/null -w "ChatGPT-User: %{http_code}\n" \
-A "ChatGPT-User/1.0" "https://YOURDOMAIN.com/?cb=$(date +%s)"
curl -s -o /dev/null -w "ClaudeBot: %{http_code}\n" \
-A "ClaudeBot/1.0" "https://YOURDOMAIN.com/?cb=$(date +%s)"
curl -s -o /dev/null -w "PerplexityBot: %{http_code}\n" \
-A "PerplexityBot/1.0" "https://YOURDOMAIN.com/?cb=$(date +%s)"Read the results plainly: 200 means that crawler can read you. 403 means it’s blocked at the edge — robots.txt never even enters the conversation. (The ?cb= bit busts caches so you’re seeing a live answer.) Test your homepage and a deep page or two; rules are sometimes scoped oddly. And run the same URLs with a normal browser user-agent as a control — if everything 403s, you have a different problem.
Fixing it in Cloudflare
If you saw 403s, log in to the Cloudflare dashboard for the domain and work through, in order:
- Security → Bots (or Security → Settings, depending on your dashboard version). Find the AI-bot controls. Newer accounts have a per-crawler policy panel — set the crawlers you want (at minimum OAI-SearchBot and ChatGPT-User, ideally all the majors) to Allow.
- Hunt for the legacy rule. If a “Block AI bots” managed rule or template exists — often scoped “Block on all pages” — it can override the newer panel. Set it to “Do not block” or remove it. This exact override was the root cause in our audit.
- Check for scheduled behavior changes. Cloudflare has migrated these features more than once, and migration prompts sometimes default to re-blocking at a future date. If you’re offered a choice about what happens “when this feature is deprecated,” choose to keep AI crawlers allowed.
- Leave your other protections alone. Allowing named AI crawlers does not mean turning off your WAF, bot-fight mode for malicious traffic, or rate limiting. This is a scalpel adjustment, not disarmament.
Then re-run the five commands. In our client’s case, every crawler flipped from 403 to 200 within seconds of the fix — no deploy, no DNS, no waiting.
Should you maybe keep them blocked?
It’s a fair question with a real trade-off. If your business model is selling the content itself — journalism, stock imagery, paid research — blocking training crawlers like GPTBot is a defensible choice. But for a service business, your website exists to make you findable and chosen. The retrieval crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot) are how AI assistants read and cite you when a potential customer asks. Blocking them is like unlisting your phone number to stop telemarketers — technically effective, commercially self-destructive. Block deliberately or allow deliberately; just don’t let a default decide for you.
Access is step one, not the whole game
Once crawlers can read you, what they find has to be worth reading: entity schema that states plainly who and where you are, content that answers real questions, and consistent facts about your business across the web. That’s the larger discipline of generative engine optimization, and access is merely its front door. If you’d like the whole thing checked properly, an AI-visibility audit is part of our SEO & performance service — it takes us a day, and the crawler test above is literally its first step.
Frequently asked questions
Doesn't robots.txt control whether ChatGPT can read my site?
Only partly. Well-behaved crawlers honor robots.txt — but a firewall block happens before robots.txt matters. A site can allow every AI bot in robots.txt and still serve them all 403s at the edge. Always test the actual HTTP response.
Which AI crawlers actually matter for being recommended?
For appearing in answers: OAI-SearchBot and ChatGPT-User (OpenAI’s search index and live fetcher), PerplexityBot, ClaudeBot, and Google’s crawlers for its AI features. GPTBot is the training-data crawler — blocking it affects future model knowledge, not live retrieval.
Will allowing AI crawlers hurt my Google SEO?
No. AI-crawler policies and Google’s ranking systems are unrelated; Googlebot has its own rules. Allowing AI crawlers simply adds another set of readers to the audience your site already serves.
I'm not technical — can I still run the check?
If you can paste five commands into a terminal, yes — copy the block above and replace YOURDOMAIN. If that’s a bridge too far, any web developer can run it in minutes, or we’ll do it as part of an audit.