Unverified50% confidenceFactExact time
robots.txt was originally designed to manage search indexing rather than AI training, and it lacks the ability to distinguish training uses, so blocking training often means blocking search indexing at the same time.
1
Sources
50%
Confidence
Long-term
Relevance
10/30/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe robots.txt protocol relies on voluntary compliance, has no legal binding force, and was originally designed without anticipating the scenario of large-scale language model training.78% similarUnverifiedTraditional robots.txt syntax only describes 'whether crawling is allowed' and does not express downstream uses such as 'whether crawled content may be used for training'.77% similarUnverifiedTraditional robot control relies on reinforcement learning or preset programs, requiring separate training for each specific task, with extremely poor generalization ability70% similarUnverifiedPrivileged-state designs in robot simulation can make training easier to converge, but fail to reflect the real-world situation in which robots can only perceive their environment through cameras.70% similarUnverifiedEmerging AI crawlers such as GPTBot, CCBot, and Amazonbot have flooded the web to collect training data in bulk, and some crawlers do not strictly comply with robots.txt restrictions or bypass filtering rules by disguising their User-Agent.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1118178API
curl https://kongchang.com/api/v1/knowledge/claims/1118178MCP
get_claim(id=1118178)