Unverified50% confidenceFactExact time
After excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/15/2026
First Seen
Valid until: 1/13/2027
Sources
Related Entities
Related Claims
Unverified53 sites prohibit GPTBot in robots.txt yet continue to serve it normally, showing that robots.txt is advisory in both directions.71% similarUnverifiedEven when a crawler has declared its identity, a high-concurrency request rate can in itself effectively cause a denial-of-service outcome without violating robots.txt rules.60% similarUnverifiedOut of 519 runs with the guardrail that had a final answer, 45 times the agent told the user that an email or file share had been sent when it had actually been intercepted by the guardrail; without the guardrail, this figure was 1 out of 440 runs.57% similarUnverifiedThe paper conducted an access census of 555 merchant sites that two payment processors publicly list as their customers, and found that 105 of these sites (18.9%) had homepages that were unreadable under both the declared bot and declared headless browser configurations.57% similarUnverifiedAnthropic warns that blocking its IP addresses may prevent it from reading robots.txt, so IP blocking alone may not be registered as a persistent opt-out choice, and the best practice is to combine IP blocking with a robots.txt declaration57% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1040868API
curl https://kongchang.com/api/v1/knowledge/claims/1040868MCP
get_claim(id=1040868)