Unverified50% confidenceCautionExact time
If a website uses a CDN or reverse proxy, it is necessary to confirm that the robots.txt in the root directory is not overridden or rewritten by the cache layer.
1
Sources
50%
Confidence
Long-term
Relevance
10/30/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIf content must be kept confidential, it should be placed behind a login, because robots.txt cannot block everything.69% similarUnverifiedWhen scraping web pages, you should comply with the site's robots.txt and terms, and avoid high-frequency requests to small sites.66% similarUnverifiedrobots.txt is publicly readable, and the list of blocked User-Agents published by operators can actually be used by crawlers to disguise themselves as other identities.66% similarUnverifiedMicrosoft explicitly states that some external bots will not be detected, while some real people will be mislabeled; Zoom also admits that real users flagged by its Web App protection can still join by logging in.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1117565API
curl https://kongchang.com/api/v1/knowledge/claims/1117565MCP
get_claim(id=1117565)