Claude-SearchBot
A web crawler operated by Anthropic that fetches web content for the Claude search feature.
Core Facts
Timeline (last 90 days)
Anthropic has three crawlers: ClaudeBot for training, Claude-User for user-triggered crawling, and Claude-SearchBot for improving search result quality.
A misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
Anthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
Marketers should check CDN or WAF logs to confirm that OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot receive 200 responses rather than 403.
The access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
All Facts (7)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%UnverifiedA misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
50%UnverifiedAnthropic has three crawlers: ClaudeBot for training, Claude-User for user-triggered crawling, and Claude-SearchBot for improving search result quality.
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedAnthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
50%UnverifiedMarketers should check CDN or WAF logs to confirm that OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot receive 200 responses rather than 403.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%