Unverified50% confidenceFactExact time
Anthropic has three crawlers: ClaudeBot for training, Claude-User for user-triggered crawling, and Claude-SearchBot for improving search result quality.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/16/2026
First Seen
Valid until: 1/14/2027
Sources
Related Entities
Related Claims
UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).71% similarUnverifiedGPTBot, ClaudeBot, and Common Crawl are known AI training crawlers.65% similarUnverifiedThere are two main strategies for acquiring recruitment data: automated web scraping and manual curation, with large platforms such as Indeed and ZipRecruiter using scraping technology.64% similarUnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools63% similarUnverifiedSome AI companies and search engines provide separate User-Agents for search indexing and AI training respectively, allowing webmasters to block training crawlers individually in robots.txt while retaining search indexing capability.60% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1044949API
curl https://kongchang.com/api/v1/knowledge/claims/1044949MCP
get_claim(id=1044949)