Unverified50% confidenceFactExact time
The robots protocol is the crawler access rules declared in a robots.txt file at a website's root directory, used to tell automated programs which paths may be crawled and which should be avoided; it is not a legal statute, but in most jurisdictions it is regarded as an important reference for judging whether crawling behavior is in good faith.
1
Sources
50%
Confidence
Long-term
Relevance
10/18/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe User-agent section of traditional robots.txt paired with Disallow is a widely recognized crawl control mechanism.74% similarUnverifiedThe industry is discussing extending robots.txt semantics, for example clearly distinguishing different purposes such as "search crawling", "training data crawling" and "agent acting on behalf of a user", along with the IETF AIPREF working group and others developing machine-readable preference expressions.68% similarUnverifiedrobots.txt User-agent matching is independent for each entry: a crawler first looks for the rule group whose name exactly matches its own UA, and only falls back to the User-agent: * wildcard rule when there is no match.67% similarUnverifiedThe core philosophy of the system is to have bots talk to bots, with humans only intervening when necessary.66% similarUnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1059042API
curl https://kongchang.com/api/v1/knowledge/claims/1059042MCP
get_claim(id=1059042)