Unverified50% confidenceConceptExact time
If retrieval-type AI crawlers (such as OAI-SearchBot and ChatGPT-User) are blocked, the content will not be cited in AI answers; but if only training-type crawlers (such as GPTBot) are blocked, the content may still be cited.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedMany AI companies' crawlers have been found to ignore the restriction directives in robots.txt, or to bypass the rules using crawlers with undisclosed identities71% similarUnverifiedMany websites' robots protocols block AI crawlers such as ClaudeBot and GPTBot, preventing these AI engines from accessing content on blocked sites.71% similarVerifiedMany AI training data crawlers tend to ignore the robots.txt protocol and rapidly scrape entire sites at extremely high concurrency, fundamentally differing from traditional search engine crawlers.70% similarUnverifiedThe same robots.txt can finely control allowing search indexing but prohibiting AI training, because different AI vendors use multiple different User-agents corresponding to different purposes.70% similarUnverifiedAI crawlers can be divided by purpose into three categories: training crawlers, search crawlers, and user-triggered fetchers, each with its own independent robots.txt identifier66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1048789API
curl https://kongchang.com/api/v1/knowledge/claims/1048789MCP
get_claim(id=1048789)