Unverified50% confidenceFactExact time
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/13/2026
First Seen
Valid until: 1/11/2027
Sources
Related Entities
Related Claims
UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools78% similarUnverifiedSome AI companies and search engines provide separate User-Agents for search indexing and AI training respectively, allowing webmasters to block training crawlers individually in robots.txt while retaining search indexing capability.77% similarUnverifiedAttackers heavily forge the User-Agents of AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, with a single IP rotating through User-Agents such as KimiBot, Qwenbot and Hunyuan within a single session while requesting the `.env` file.73% similarUnverifiedGPTBot, ClaudeBot, and Common Crawl are known AI training crawlers.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1020795API
curl https://kongchang.com/api/v1/knowledge/claims/1020795MCP
get_claim(id=1020795)