ClaudeBot
A web crawler used by Anthropic to fetch web content for training Claude models, identified by its user-agent.
Core Facts
Timeline (last 90 days)
Anthropic explicitly states that blocking ClaudeBot means the website's future content is excluded from the training dataset, i.e., blocking only takes effect for the future
At present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
Attackers heavily forge the User-Agents of AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, with a single IP rotating through User-Agents such as KimiBot, Qwenbot and Hunyuan within a single session while requesting the `.env` file.
Anthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
Many websites' robots protocols block AI crawlers such as ClaudeBot and GPTBot, preventing these AI engines from accessing content on blocked sites.
GPTBot, ClaudeBot, and Common Crawl are known AI training crawlers.
The access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
Mainstream AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google DeepMind), and Bytespider (ByteDance).
ClaudeBot is Anthropic's crawler
All Facts (11)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%Partially VerifiedClaudeBot is Anthropic's crawler
65%UnverifiedMainstream AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google DeepMind), and Bytespider (ByteDance).
60%UnverifiedAnthropic explicitly states that blocking ClaudeBot means the website's future content is excluded from the training dataset, i.e., blocking only takes effect for the future
50%UnverifiedAt present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedAttackers heavily forge the User-Agents of AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, with a single IP rotating through User-Agents such as KimiBot, Qwenbot and Hunyuan within a single session while requesting the `.env` file.
50%UnverifiedAnthropic's split logic is consistent with OpenAI's: ClaudeBot collects content that may be used for training, Claude-SearchBot is used to improve search result quality, and Claude-User only crawls pages when a user actively asks.
50%UnverifiedMany websites' robots protocols block AI crawlers such as ClaudeBot and GPTBot, preventing these AI engines from accessing content on blocked sites.
50%UnverifiedGPTBot, ClaudeBot, and Common Crawl are known AI training crawlers.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%