Unverified50% confidenceFactExact time
AI crawlers can be divided by purpose into three categories: training crawlers, search crawlers, and user-triggered fetchers, each with its own independent robots.txt identifier
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedMany AI companies' crawlers have been found to ignore the restriction directives in robots.txt, or to bypass the rules using crawlers with undisclosed identities76% similarVerifiedMany AI training data crawlers tend to ignore the robots.txt protocol and rapidly scrape entire sites at extremely high concurrency, fundamentally differing from traditional search engine crawlers.76% similarVerifiedAI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.75% similarUnverifiedCloudflare's AI crawler policy can separately block search, agent, and training crawlers, with finer granularity than robots.txt72% similarUnverifiedA 2024 study found that among known AI training crawlers, more than 35% completely disregard the restriction directives in robots.txt.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1028524API
curl https://kongchang.com/api/v1/knowledge/claims/1028524MCP
get_claim(id=1028524)