aeo
Blocking AI Crawlers Might Be Costing You Citations.
Most B2B marketing teams have never looked at their own robots.txt file, and many of the ones who have looked at it added a blanket rule to keep AI out. Blocking AI crawlers feels like a safe default when the loudest conversation about AI and content is about scraping and copyright, but for a company trying to be cited by ChatGPT, Perplexity, Gemini, or Copilot, that same rule quietly closes the one door those engines need to walk through. You cannot be named in an answer if the engine was never allowed to read the page.
Why would blocking AI crawlers hurt my citation rate?
Blocking AI crawlers hurts your citation rate because most answer engines can only cite content they, or a partner index, have actually fetched. Unlike a human visitor who can be shown an "allow cookies" banner and then read the page, a blocked crawler gets a 403 or a disallow directive and moves on permanently, with no retry and no notification to your team that anything happened. SourceRank AI audit data already shows the average B2B company is cited in fewer than 5% of relevant AI prompts, and for a meaningful share of the companies we audit, part of that gap is not weak content or missing entity signals at all. It is a robots.txt rule, a CDN setting, or a security plugin installed years before anyone on the team had heard of an AI citation, still blocking access today.
What is the difference between a training bot and a retrieval bot?
A training bot crawls your site to build or update the model itself, while a retrieval bot fetches a page in response to a specific live query so the engine can quote or summarise it in that moment. GPTBot, ClaudeBot, and Google-Extended fall into the first group. OAI-SearchBot, PerplexityBot, ChatGPT-User, and Bingbot (which underpins Copilot's index) fall into the second. This distinction matters because the two groups raise different questions. Blocking a training bot is a legitimate content-licensing decision that has little to do with whether you get cited tomorrow. Blocking a retrieval bot removes you from that engine's answers immediately, because the engine cannot quote a page it is not permitted to fetch at query time.
The problem is that most "block all AI" toggles, whether set by a security plugin, a CDN rule, or a well-meaning IT policy, do not distinguish between the two. They block both groups with a single switch, trading a training-data decision your legal team may care about for a citation outcome your marketing team is trying to earn.
Which AI bots should a B2B company actually allow?
A B2B company trying to earn AI citations should allow the retrieval bots for every engine it wants to be cited in, and treat the training bots as a separate decision made deliberately rather than by default.
| Bot | Operator | Type | Typical robots.txt guidance for AEO |
|---|---|---|---|
| GPTBot | OpenAI | Training | Allow if you want ChatGPT's underlying model trained on your content; blocking it does not remove you from live ChatGPT search citations |
| OAI-SearchBot | OpenAI | Retrieval | Allow. This is the crawler behind ChatGPT's live search and citation feature |
| ChatGPT-User | OpenAI | Retrieval (user-triggered) | Allow. Fetches a page when a user's live prompt asks ChatGPT to browse it |
| PerplexityBot | Perplexity | Retrieval | Allow. Blocking this is the single most common cause of zero Perplexity citations we see in audits |
| ClaudeBot | Anthropic | Training | Optional; treat as a licensing decision, not a citation one |
| Claude-SearchBot | Anthropic | Retrieval | Allow if you want Claude's search and citation features to reach the page |
| Google-Extended | Training/AI features opt-out | Blocking this affects Gemini and Google's AI features, not standard Search ranking | |
| Bingbot | Microsoft | Retrieval (also classic search) | Allow. Copilot draws on Bing's index, so blocking Bingbot affects Copilot citation too |
How do you check whether your site is already blocking AI crawlers?
Run through this order before you touch any content, because a blocked crawler makes every other AEO fix invisible to the engine.
- 1. Read the live robots.txt file. Open yourdomain.com/robots.txt directly and look for a `Disallow: /` line under any of the user-agents above, or a blanket rule with no user-agent specified at all.
- 2. Check your CDN or WAF bot-management rules. Cloudflare, Akamai, and similar platforms often ship an "AI bot" or "AI scraper" blocking category that can be switched on separately from robots.txt, and it overrides robots.txt when active.
- 3. Check for a security plugin doing the same thing. WordPress and other CMS security plugins added blanket AI-blocking toggles during 2024 and 2025 in response to scraping concerns, frequently on by default after an update.
- 4. Search your server logs for the user-agent strings. A clean log showing regular 200 responses to GPTBot, PerplexityBot, OAI-SearchBot, and ChatGPT-User confirms access; a run of 403s or no requests at all confirms a block somewhere in the chain.
- 5. Re-test after any change. Crawlers do not retry on a fixed schedule you control, so a fix does not show up in citations immediately. Track your prompt set against your visibility score over the following weeks rather than expecting an instant jump.
Can a CDN or WAF block AI crawlers even when robots.txt allows them?
Yes, and this is the check most teams skip. robots.txt is a voluntary instruction that well-behaved crawlers honour, but a CDN, firewall, or bot-management layer sits in front of it and can reject a request before robots.txt is ever consulted. A site can have a perfectly permissive robots.txt file and still be invisible to every retrieval bot because a security layer added months earlier is silently returning a 403. This is why step two in the audit above matters as much as step one: fixing the file without checking the infrastructure in front of it leaves the actual block in place.
Should you block AI training while still allowing AI citation?
Yes, this is a reasonable and increasingly common position, and the two decisions are genuinely separable in a correctly written robots.txt file. You can disallow GPTBot and ClaudeBot specifically, keeping your content out of general model training, while explicitly allowing OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, and Bingbot, keeping your content eligible for live citation in each engine's answers. Writing separate user-agent blocks for training crawlers and retrieval crawlers, rather than one rule covering "AI" as a single category, is the technical step that makes this distinction actually work. It is also exactly the kind of on-site fix that sits inside SourceRank AI's services, alongside the entity and structured data work that determines whether a page gets quoted once it is reachable.
Key Terms Glossary
Frequently asked questions
How do I know if my robots.txt is blocking AI crawlers?
Open yourdomain.com/robots.txt in a browser and check for a `Disallow: /` rule listed under GPTBot, PerplexityBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, or Claude-SearchBot, or a blanket disallow with no user-agent named, which applies to every crawler including these.
Does blocking GPTBot stop me from appearing in ChatGPT?
Not directly. GPTBot is OpenAI's training crawler, so blocking it affects whether your content trains future models but does not remove you from live ChatGPT search results, which run through the separate OAI-SearchBot and ChatGPT-User crawlers.
Will Cloudflare block AI crawlers by default?
Not by default on every plan, but many accounts have an AI bot-blocking category available in the dashboard that a security-conscious admin may have switched on separately from robots.txt, so it is worth checking even if your robots.txt file looks permissive.
Is it bad for SEO to block AI crawlers?
Blocking most AI crawlers does not affect classic Google Search rankings, since Googlebot is separate from Google-Extended. It specifically affects whether AI answer engines can cite you, which is a different visibility channel from ranking on a results page.
Can I allow Perplexity but block ChatGPT's training crawler?
Yes. robots.txt supports per-bot user-agent rules, so you can write a specific allow rule for PerplexityBot and a specific disallow rule for GPTBot in the same file without either affecting the other.
How often should I check for AI crawler blocks?
Check whenever you change CDN providers, install or update a security plugin, or notice a drop in AI citation rate with no corresponding content change, since any of those can silently introduce a block that a one-time check will not catch later.
Does an AI crawler block show up in normal web analytics?
Usually not. Most AI crawlers do not execute JavaScript, so tools like Google Analytics that rely on a JavaScript tag will not register their visits at all, blocked or not. Server log analysis is the reliable way to confirm access.
What should I do first if I think I am blocked?
Run the five-step check in this guide, starting with the live robots.txt file and moving to CDN and security-plugin settings, then confirm the fix against your free visibility score before assuming any content or schema work will move your citation rate. If the audit turns up more than a crawler block, contact us or see how it works and pricing for what a full fix looks like.