Unverified50% confidenceOpinionExact time
There is no reciprocal relationship between the operators of AI training crawlers and websites, because they do not rely on search results for traffic, and therefore are not bound by the industry tacit understanding behind robots.txt.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/13/2026
First Seen
Valid until: 1/11/2027
Sources
Related Entities
Related Claims
UnverifiedAI training crawlers generally do not obey the robots.txt protocol, nor do they self-restrain their request frequency, with crawling behavior approaching a DDoS attack.79% similarUnverifiedThe web search capabilities that most AI agents inherit directly from underlying frameworks or default toolchains are not sufficiently capable.70% similarUnverified不应给AI智能体超出任务所需的访问权限，负责预约健身课的智能体不应拥有触及后台数据库的能力67% similarUnverifiedSecurity boundaries do not automatically tighten just because the operator is an AI; they only become looser because the process does not distinguish between 'people' and 'agents.'66% similarUnverifiedThe author suggests blocking AI crawlers only when there is a clear reason, and lists three common situations: not wanting content to be used for model training, crawler traffic causing server load or bandwidth costs, and content being paid or licensed content.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1019859API
curl https://kongchang.com/api/v1/knowledge/claims/1019859MCP
get_claim(id=1019859)