Should AI Crawlers Access Your Website? Training and Search Require Separate Decisions

Many websites added Disallow: / for GPTBot back in 2023 and never looked back. Some even blocked every AI user agent they could find. There's nothing wrong with that in itself, but many people made that decision without knowing that different AI crawlers serve different purposes. Block a training crawler and your content won't enter the training set; block a search crawler and you may vanish entirely from AI answers. This article separates the two categories to help you make a deliberate trade-off.

One company, multiple crawlers

OpenAI, Anthropic, and Perplexity all operate more than one user agent. Each company's official documentation separates them by purpose, and they can be controlled separately in robots.txt.

OpenAI's four crawlers

• GPTBot: Crawls content that may be used to train OpenAI's generative AI foundation models. Disabling it is a declaration that your content should not be used for training.
• OAI-SearchBot: Makes websites appear in ChatGPT's search features. OpenAI states that sites opting out of OAI-SearchBot won't appear in ChatGPT search answers but may still appear as navigational links.
• ChatGPT-User: Visits pages for certain user actions in ChatGPT and custom GPTs. OpenAI says it's not used for automated crawling, and because actions are user-initiated, robots.txt rules may not apply.
• OAI-AdsBot: Verifies pages submitted as ads to ChatGPT and is not used to train models.

OpenAI emphasizes that each setting is independent: you can allow OAI-SearchBot while disallowing GPTBot. It also notes that search-related robots.txt changes take about 24 hours to take effect.

Anthropic's three crawlers

According to Anthropic's help center:

• ClaudeBot: Collects web content that may be used for model training. Restricting it means future content should be excluded from training datasets.
• Claude-User: Crawls pages when users ask Claude questions. Disabling it may reduce your visibility in user-initiated web searches.
• Claude-SearchBot: Browses the web to improve search result quality. Disabling it blocks the search optimization index and may reduce your exposure in user search results.

Anthropic states that its crawlers respect robots.txt and support the non-standard Crawl-delay directive. It also warns that blocking by IP address may be unreliable because it prevents Anthropic from reading your robots.txt.

Perplexity's two crawlers

• PerplexityBot: Surfaces and links websites in Perplexity search results. Perplexity says it's not used to crawl content for AI foundation models.
• Perplexity-User: Supports user actions. Perplexity says it generally ignores robots.txt because crawling is requested by the user.

Why retrieval-augmented generation (RAG) makes search crawlers critical. Many people imagine "AI answers" as a model writing from memory, but current mainstream products, when answering time-sensitive or fact-specific questions, typically retrieve a batch of web pages in real time first, then hand the retrieved content to the model to organize the language. This step is retrieval-augmented generation. Which sources it can surface and link to depends on the pool of pages it has previously crawled and indexed. Crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot exist to fill that pool. So blocking a search crawler isn't just "not letting it read"—it keeps your pages out of the candidate set entirely, leaving the model nothing to cite. This also explains why blocking a training crawler versus a search crawler produces completely different consequences: the former affects the parameters of future models, the latter affects your exposure right now.

Training crawlers and search crawlers are two different decisions

A more useful framework for thinking about this:

Training crawlers (GPTBot, ClaudeBot) feed future models. Blocking them is about your control over how content is used, with no documented direct impact on whether you appear in real-time search answers.

Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) power the retrieval side. Block them and you're out of the source pool these products can surface and link to.

User-triggered crawling (ChatGPT-User, Claude-User, Perplexity-User) represents a specific person asking a specific question. Some vendors say robots.txt may not apply.

If your goal is to be recommended by AI assistants, allow search crawlers. Whether to allow training crawlers is a separate business judgment.

How robots.txt works and its limits. robots.txt is a plain text file in your website's root directory; crawlers should read it before crawling to decide whether to access a given path. It relies on voluntary compliance, essentially a convention rather than an enforced lock. Rules are grouped by user-agent, each group starting with one or more User-agent lines followed by Allow or Disallow paths; more specific groups (exact name matches) override the wildcard group *. Paths support prefix matching, and Disallow: / means rejecting the entire site. Note that robots.txt only governs "automated crawling" and has weak power over crawls triggered by a real person's click—which is why "user-triggered crawling" is called out separately here. It also isn't shared between subdomains; each host needs its own copy.

How to decide: three questions

1. Do we want AI assistants to recommend us? For most businesses selling something, the answer is yes. Allow search crawlers.

2. Is the content itself the product? Publishers, paid research firms, and licensed content sites have reason to block training crawlers—the content is what they're selling.

3. Can you accept your content being used for training? For software companies with public marketing pages, many teams believe the benefit of being "known" to models outweighs the cost; others disagree. Both are valid. Note that blocking training crawlers only affects future crawls, not data already collected.

There's no universally correct answer, only the one you choose deliberately after being informed.

Sample robots.txt configurations

Each group starts with a User-agent line, and rules apply only to the agent specified in that group.

Allow all AI crawlers

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

If no rule blocks these agents, they are typically allowed by default. Explicit listing isn't required but makes intent clearer.

Appear in AI search, opt out of training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

This aligns with each company's own descriptions: block training crawlers, allow search crawlers. Note this is based on your interpretation of official documentation. Vendors may adjust crawler policies, so periodic review is advisable.

Block only private areas

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /account/
Disallow: /internal/

Stacking multiple User-agent lines under one set of rules is valid. Most websites should do this rather than blanket blocking everything.

Common mistakes

Blanket blocking with the wildcard User-agent: *: This blocks all crawlers without a more specific group, including all the search crawlers above. Check whether you have a wildcard rule.

Blocking at the firewall level: CDN or WAF bot rules may return 403 to crawlers that robots.txt allows. Perplexity and OpenAI both publish IP ranges for allowlisting; Anthropic explicitly asks you not to rely on IP blocking to opt out.

Misspelling user-agent names: The correct names are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User. Misspelled names won't match any crawler and will fail silently.

Assuming robots.txt blocks everything: It's a request, not a lock. Vendors say their crawlers comply, but some user-triggered crawls may not. If content must stay confidential, put it behind a login.

Forgetting subdomains: robots.txt applies per host. Anthropic reminds you that each subdomain must be configured separately.

Why CDN and WAF bot management so easily do harm by accident. Many sites enable "Bot Fight Mode" or custom WAF rules on Cloudflare or similar CDNs. These rules block requests at the edge, so requests never reach the origin server and thus can't read your robots.txt. The result: robots.txt says Allow, but the logs show no corresponding user-agent and pages keep getting rejected. The article suggests allowlisting by public IP ranges, but note that IP ranges change and need periodic syncing; Anthropic explicitly opposes relying on IP blocking to opt out because it also blocks its ability to read your robots.txt. The order for troubleshooting these issues should be: first allow at the edge layer, then verify robots.txt can be read normally, and only then check server logs to confirm whether crawlers actually arrive.

Test the rules you've set

Don't trust an untested file. You can use an AI crawler access checker to read your robots.txt and see which AI user agents are allowed or blocked. Run it after every change, and again after CDN or firewall updates. Then check your server logs for the user-agent names above—if they never appear despite being allowed, something between the internet and your server is getting in the way.

Also consider llms.txt

Robots.txt tells crawlers what they may crawl; llms.txt is a proposed plain text file for guiding language models to your most useful pages. Treat it as a low-cost supplement, not a universal fix. If you want to try it, you can use an llms.txt generator to create a starter file from your website.

The positioning and limits of llms.txt. llms.txt is a community-proposed convention not yet formally adopted by mainstream vendors: place a Markdown-format plain text file in your site's root directory, using headings and link lists to point out the pages most valuable to language models (such as product descriptions, docs, and pricing), so models don't get lost in navigation and marketing noise. Its role differs from robots.txt—the latter governs "can it crawl," the former governs "what should it look at." Since there's no enforced standard, whether it's read depends entirely on each implementation, so for now it can only be a nice-to-have supplement; it can't replace robots.txt's access control, nor guarantee citation. If your site's content structure is complex, ensuring a clear sitemap and semantic pages is often more effective than adding llms.txt.

Then measure the results

Editing robots.txt is just an input; what you care about is the output: do AI assistants mention and cite you? Once your rules are set, track it. Pineprompt can run your prompts across ChatGPT, Gemini, Perplexity, Claude, Google AI Overviews, AI Mode, Grok, and Copilot, reporting where you appear and which sources are cited. Plans start at $99 per month and include all engines. See the pricing page for details.

Originally published at pineprompt.com.

One-line Summary

AI crawlers fall into three types—training, search, and user-triggered—which robots.txt can control separately, so blocking decisions should be made independently.

Paragraph Summary

OpenAI, Anthropic, and Perplexity each operate multiple user agents serving different purposes: training, search, and user-triggered crawling, all of which robots.txt can control separately. Blocking training crawlers only affects future crawls, not data already collected; blocking search crawlers removes your site from the source pool for AI answers and may make you vanish entirely from responses. The article proposes a framework: if you want AI assistants to recommend you, allow search crawlers; if your content itself is the product, you have reason to block training crawlers. It provides three sample configurations, five common mistakes (blanket wildcard blocking, firewall blocking, misspelled names, assuming robots.txt is all-powerful, ignoring subdomains), and stresses that after setup you must verify with checker tools and server logs. llms.txt can be a low-cost supplement, but ultimately you should track whether AI assistants actually cite you.