Expired60% confidenceEventExact time
In 2024, Wired and Forbes revealed that Perplexity's crawler engaged in behaviors such as bypassing the robots.txt protocol and scraping paywalled content.
2
Sources
60%
Confidence
Medium-term (~90 days)
Relevance
7/19/2026
First Seen
Valid until: 10/17/2026(expired)
Sources
Related Claims
Unverifiedrobots.txt is an access-control mechanism that tells crawlers what can and cannot be crawled; llms.txt is a content navigation tool that proactively recommends core content worth reading to models64% similarVerifiedGooglebot determines the scope of crawlable pages by following each website's robots.txt protocol; robots.txt is a text file stored in the root directory of a website used to declare crawlable or blocked pages to web crawlers.61% similarUnverifiedThe default robots.txt generated by many hosting platforms or templates blanket-Disallows all crawlers, making content visible to Googlebot but completely invisible to AI engines60% similarVerifiedrobots.txt is a plain text file placed in a website's root directory that tells web crawlers which pages can be crawled and which should be avoided59% similarUnverifiedrobots.txt 和 sitemap.xml 作为网站向机器暴露元信息的早期实践，可被视为 WebMCP 的精神前身59% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/560929API
curl https://kongchang.com/api/v1/knowledge/claims/560929MCP
get_claim(id=560929)