Unverified50% confidenceFactExact time
53 sites prohibit GPTBot in robots.txt yet continue to serve it normally, showing that robots.txt is advisory in both directions.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/15/2026
First Seen
Valid until: 1/13/2027
Sources
Related Entities
Related Claims
UnverifiedAfter excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.71% similarUnverifiedrobots.txt is a 'gentleman's agreement' that relies on the crawler's voluntary compliance and has no enforcement power64% similarUnverifiedrobots.txt is publicly readable, and the list of blocked User-Agents published by operators can actually be used by crawlers to disguise themselves as other identities.63% similarUnverifiedEven when a crawler has declared its identity, a high-concurrency request rate can in itself effectively cause a denial-of-service outcome without violating robots.txt rules.62% similarUnverifiedEnterprise protection measures include blocking sensitive paths in robots.txt, enabling IP whitelisting for the backend, and using non-standard paths to replace the default /admin entry point.61% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1040870API
curl https://kongchang.com/api/v1/knowledge/claims/1040870MCP
get_claim(id=1040870)