Claude-User
Anthropic's real-time user-agent crawler that fetches page content on behalf of a Claude user when that user requests browsing of a web page.
Core Facts
Timeline (last 90 days)
Anthropic says that disallowing Claude-User will prevent Claude from retrieving pages for user queries, but OpenAI and Perplexity say their own user-triggered fetchers may not obey robots.txt
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
Anthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
The access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
All Facts (5)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%UnverifiedAnthropic says that disallowing Claude-User will prevent Claude from retrieving pages for user queries, but OpenAI and Perplexity say their own user-triggered fetchers may not obey robots.txt
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedAnthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%