Unverified85% confidenceFactExact time
Major media organizations like The New York Times and Reuters were among the first to block AI crawlers in their robots.txt files
1
Sources
85%
Confidence
Medium-term (~90 days)
Relevance
8/2/2026
First Seen
Sources
Only 8.9% of Websites Block AI Crawlers, Yet 94.8% Have Never Been Cited in AI Answers
Related Entities
Related Claims
Partially Verifiedrobots.txt is a plain text file placed in a website's root directory that tells web crawlers which pages can be crawled and which should be avoided60% similarUnverifiedGooglebot determines the scope of crawlable pages by following each website's robots.txt protocol; robots.txt is a text file stored in the root directory of a website used to declare crawlable or blocked pages to web crawlers.59% similarUnverifiedSeveral companies including Google and Adobe have encountered similar privacy controversies when launching AI image features.57% similarUnverified谷歌旗下Waymo与Uber围绕自动驾驶技术展开过由员工跳槽引发的商业秘密诉讼，最终以巨额和解落幕56% similarVerifiedrobots.txt is an industry convention established in 1994 by the Robots Exclusion Protocol (REP) and is not a mandatory legal regulation, but mainstream search engines such as Google and Baidu strictly adhere to it56% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/679742API
curl https://kongchang.com/api/v1/knowledge/claims/679742MCP
get_claim(id=679742)