Claude-SearchBot
A web crawler operated by Anthropic that fetches web content for the Claude search feature.
Core Facts
Timeline (last 90 days)
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
Anthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
Marketers should check CDN or WAF logs to confirm that OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot receive 200 responses rather than 403.
The access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
All Facts (5)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedAnthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
50%UnverifiedMarketers should check CDN or WAF logs to confirm that OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot receive 200 responses rather than 403.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%