AI Crawlers Explained: Which Ones to Block and Which to Allow

AI companies' crawlers are reading your website in different ways. Blocking them as a single group can cost you far more than you expect — you would not only opt out of model training, but also disappear from AI search answers.

According to the AI crawler guide published by CheckSEO, these crawlers can be divided into three categories by purpose, each with its own robots.txt identifier, so blocking AI is never an all-or-nothing choice.

Three Types of AI Crawlers, Three Responsibilities

Training crawlers collect content to train future models, represented by OpenAI's GPTBot, Anthropic's ClaudeBot, and Common Crawl's CCBot.

Search crawlers index pages so that AI answers can cite and link back to your content, including OAI-SearchBot, Claude-SearchBot, and PerplexityBot.

User-triggered fetchers load pages in real time when someone asks a question, such as ChatGPT-User and Perplexity-User.

Two easily confused entries need separate clarification: Google-Extended and Applebot-Extended are not crawlers at all, but control tokens in robots.txt that determine whether content crawled by Googlebot and Applebot can be used for Gemini training or Apple model training. They will never appear in server logs.

robots.txt is a plain-text protocol file stored in a website's root directory, with origins dating back to 1994. Through two sets of directives — User-agent (specifying the target crawler) and Disallow/Allow (specifying path rules) — it tells crawlers which paths can be crawled and which cannot. It is a "gentlemen's agreement": mainstream search engine crawlers actively read and obey it, but the protocol itself provides no technical enforcement. Understanding robots.txt's matching logic is a prerequisite for understanding the configuration later in this article — a crawler first looks for a User-agent group matching its own name, and only falls back to the wildcard group User-agent: * if there is no match; once it finds a dedicated group, the rules in the wildcard group no longer apply to it at all.

User-agent is a string field in HTTP request headers, used by crawlers to identify themselves. For example, GPTBot includes an identifier like Mozilla/5.0 AppleWebKit/537.36; compatible; GPTBot/1.0 when making requests. Both servers and robots.txt rely on this string to identify who is visiting. However, because the string is written by the client itself, the cost of spoofing is almost zero, which is why blocking based on user agent alone is unreliable and must be combined with IP verification — the "verify visitor identity" mentioned later is aimed precisely at this weakness.

What Does Blocking Each Category Actually Cost?

Blocking training crawlers is the lowest-cost option. OpenAI records GPTBot and OAI-SearchBot as independent settings, so if you disallow GPTBot but allow OAI-SearchBot, you are still eligible to appear in ChatGPT search. Note that this kind of blocking only applies to the future — Anthropic states explicitly that blocking ClaudeBot means a website's future content is excluded from its training datasets.

Blocking search crawlers means opting out of AI answers. OpenAI says websites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, although they may still appear as navigational links.

Blocking user-triggered fetchers is the hardest to predict. Anthropic says disallowing Claude-User prevents Claude from retrieving pages for user queries, but OpenAI and Perplexity both say their fetchers act on behalf of users and may not comply with robots.txt.

Google's situation is more unusual. Google states explicitly that Google-Extended does not affect a website's inclusion in Google Search and is not a ranking signal. Because AI Overviews and AI Mode are part of search functionality, blocking Google-Extended will not make you disappear from these features.

Why is blocking training crawlers the "lowest-cost" option? The key lies in the time dimension of crawling behavior. Training crawlers (such as GPTBot and CCBot) crawl web pages offline and in batches, storing content in datasets to train future models; they do not visit your site in real time at the moment a user asks a question. Therefore, blocking them only keeps your content out of the training corpus for the next generation of models, with almost no impact on whether users can find you through AI now or whether you can be cited in real time. Search crawlers and user-triggered fetchers, by contrast, directly determine whether "you are in the AI answer right now," and the immediate visibility loss from blocking them is much greater. This also explains why the exit costs of the three crawler categories show a clear gradient: training < search < user-triggered.

"Only applies to the future" means the irreversibility of training data: once a model has finished training on your content, its parameters are fixed, and later robots.txt blocking cannot make the model "forget." So blocking training crawlers is a preventive measure, not a remedy — it protects future versions of the model from absorbing your content.

Copy-and-Paste robots.txt Rules

The following is the most common middle-ground configuration — block training while preserving visibility in AI search:

# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# Preserve visibility in AI search answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

If you want to allow everything, the configuration is much simpler:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

The file needs to be placed in the root directory of every host that needs to be covered, because each subdomain needs its own robots.txt. Changes take effect after about a day.

Here is an easy pitfall to fall into: when a crawler finds a rule group specifically naming it, it completely ignores the User-agent: * group. Therefore, you need to copy any path rules you rely on (such as Disallow: /admin/) into every named group.

robots.txt Is a Request, Not Enforcement

Technically, nothing prevents a bot from ignoring robots.txt. If you need hard blocking, it must be enforced at the edge layer:

1. Verify visitor identity. User agent strings are extremely easy to spoof. OpenAI, Anthropic, Perplexity, Common Crawl, and Apple all publish their IP addresses, so you need to match both the name and the source before deciding whether to trust or block.
2. Block by behavior at the CDN or firewall layer. Cloudflare's AI crawler policies can separately block search, agent, and training crawlers. Other firewalls can also match user agent plus IP ranges.
3. Still keep robots.txt. Anthropic warns that blocking its IP addresses may prevent it from reading your robots.txt, so IP blocking alone may not register as a persistent opt-out choice.

A CDN (Content Delivery Network) is a layer of distributed proxy servers between users and the origin server, while a WAF (Web Application Firewall) is deployed at the traffic entry point and filters requests according to rules. Both can complete blocking before requests reach the origin, which is exactly what "edge-layer enforcement" means — because robots.txt depends entirely on crawler self-discipline, when true hard blocking is needed, the decision can only be made at such edge nodes. Take Cloudflare as an example: its AI crawler policies classify crawlers into search, agent, training, and other categories, and blocking can be enabled separately, giving finer granularity than robots.txt.

IP verification is the core method of edge blocking. OpenAI, Anthropic, Perplexity, and other companies publish the IP ranges used by their crawlers, allowing administrators to check whether request sources are truly official, thereby distinguishing real crawlers from spoofed traffic. But there is a trap here: if you only block IPs without keeping robots.txt, crawlers may not even be able to read your robots.txt, and instead cannot register "I reject you" as a formal, persistent opt-out record. Therefore, the best practice is to combine "IP blocking + robots.txt declaration."

A Few Details Worth Noticing

PerplexityBot's robots.txt compliance status is marked as "disputed" — Perplexity claims to comply, but outsiders have questioned this. For Bytespider (ByteDance), no publicly released compliance policy was found.

Among Meta's several tokens, Meta-ExternalFetcher explicitly states that it may bypass robots.txt, while Meta-WebIndexer complies.

To quickly check how open your website is to various AI crawlers, you can use the free AI Crawler Access Checker provided by CheckSEO. It fetches a domain's robots.txt and reports allow, block, or partial restriction for 16 user agents one by one, along with the specific rule behind each judgment, with no registration required.

One-line Summary

AI crawlers fall into training, search, and user-triggered categories, so blocking requires precise trade-offs rather than a blanket approach

Paragraph Summary

AI companies' crawlers are reading websites in different ways. Blocking AI crawlers as a whole would cause a site to opt out of both model training and AI search answers. According to CheckSEO's AI crawler guide, these crawlers fall into three categories by purpose: training crawlers that collect training corpora (GPTBot, ClaudeBot, CCBot), search crawlers that index for citations (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered fetchers that crawl in real time when users ask questions (ChatGPT-User, Perplexity-User). The article notes that blocking training crawlers has the lowest cost and does not affect AI search visibility, blocking search crawlers is equivalent to opting out of AI answers, and user-triggered fetchers are the hardest to predict. Google-Extended and Applebot-Extended are not crawlers but control tokens, and do not affect traditional search inclusion. robots.txt is only a request, not enforcement. Hard blocking must be implemented at the CDN or firewall layer combined with IP verification.