Unverified50% confidenceFactExact time
AI training crawlers generally do not obey the robots.txt protocol, nor do they self-restrain their request frequency, with crawling behavior approaching a DDoS attack.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/13/2026
First Seen
Valid until: 1/11/2027
Sources
Related Entities
Related Claims
UnverifiedThere is no reciprocal relationship between the operators of AI training crawlers and websites, because they do not rely on search results for traffic, and therefore are not bound by the industry tacit understanding behind robots.txt.79% similarUnverifiedThe behavior of AI crawlers is essentially application-layer (L7) resource exhaustion, and because their request patterns more closely resemble those of legitimate users, they are harder to identify and mitigate through traffic characteristics than traditional DDoS attacks.71% similarUnverifiedThe author suggests blocking AI crawlers only when there is a clear reason, and lists three common situations: not wanting content to be used for model training, crawler traffic causing server load or bandwidth costs, and content being paid or licensed content.67% similarUnverifiedAgainst distributed AI crawlers, the defensive effectiveness of rate limiting and IP blocking is greatly diminished.65% similarUnverifiedBecause malicious payloads are indistinguishable in appearance from legitimate task instructions, traditional worm defenses based on signature detection or traffic analysis are almost entirely ineffective against AI agent worms.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1019858API
curl https://kongchang.com/api/v1/knowledge/claims/1019858MCP
get_claim(id=1019858)