Unverified60% confidenceFactExact time
robots.txt is a 'gentleman's agreement' that relies on the crawler's voluntary compliance and has no enforcement power
2
Sources
60%
Confidence
Long-term
Relevance
8/2/2026
First Seen
Sources
Only 8.9% of Websites Block AI Crawlers, Yet 94.8% Have Never Been Cited in AI Answers
Related Entities
Related Claims
Unverifiedrobots.txt is publicly readable, and the list of blocked User-Agents published by operators can actually be used by crawlers to disguise themselves as other identities.67% similarUnverifiedEven when a crawler has declared its identity, a high-concurrency request rate can in itself effectively cause a denial-of-service outcome without violating robots.txt rules.60% similarUnverifiedWhen the bot encounters a problem it cannot resolve, it collects the user's email and forwards the issue to a human agent for handling59% similarUnverifiedThe waiting room is the preferred means of intercepting unknown bots, because it changes 'default trust' to 'default verification'—even if a bot successfully initiates a join request via calendar or link, it must be manually admitted before it can enter.58% similarUnverifiedEffective technical means for creators to protect content include declaring crawling rules via the robots.txt protocol, server-side rendering combined with access frequency limiting, and setting login walls for high-value content.57% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/679738API
curl https://kongchang.com/api/v1/knowledge/claims/679738MCP
get_claim(id=679738)