robots.txt
A plain-text file protocol placed in a website's root directory that controls crawler access to pages through User-Agent fields and Disallow/Allow directives. Introduced in 1994, it operates as a voluntary, honor-based convention without legally binding enforcement mechanisms.
Core Facts
The robots.txt protocol was first proposed by Dutch engineer Martijn Koster in 1994
The robots.txt protocol was proposed by internet pioneer Martijn Koster in 1994 and is a plain text file placed in the root directory of a website
The robots.txt protocol relies entirely on voluntary compliance by crawlers and has no technical enforcement mechanism whatsoever.
Timeline (last 90 days)
llms.txt has a different positioning from robots.txt (which controls crawler access permissions) and sitemap.xml (which lists URLs); its goal is to let models quickly understand a site's structure and core content.
Some vendor documentation states that user-triggered AI agents may not follow robots.txt the way traditional crawlers do, because the requests are initiated by a real person.
There is no reciprocal relationship between the operators of AI training crawlers and websites, because they do not rely on search results for traffic, and therefore are not bound by the industry tacit understanding behind robots.txt.
AI training crawlers generally do not obey the robots.txt protocol, nor do they self-restrain their request frequency, with crawling behavior approaching a DDoS attack.
robots.txt only controls whether crawling is allowed, not whether indexing is allowed; a URL that is disallowed from crawling may still appear in search results if it is linked from other sites (usually showing only the link without a snippet).
robots.txt User-agent matching is independent for each entry: a crawler first looks for the rule group whose name exactly matches its own UA, and only falls back to the User-agent: * wildcard rule when there is no match.
Even when a crawler has declared its identity, a high-concurrency request rate can in itself effectively cause a denial-of-service outcome without violating robots.txt rules.
There is debate over whether the robots.txt protocol has legal binding force on AI training crawlers.
The robots.txt protocol was originally established in 1994 as an industry convention by which websites inform crawlers which pages may be scraped, but it carries no legally binding force.
AI agent execution frameworks (such as LangChain, AutoGPT, or OpenAI's Assistants API) do not by default enforce checking of robots.txt, unless developers actively implement the relevant logic.
23 more timeline events
All Facts (20)
The robots.txt protocol was first proposed by Dutch engineer Martijn Koster in 1994
80%VerifiedThe robots.txt protocol was proposed by internet pioneer Martijn Koster in 1994 and is a plain text file placed in the root directory of a website
75%VerifiedThe robots.txt protocol relies entirely on voluntary compliance by crawlers and has no technical enforcement mechanism whatsoever.
70%Verifiedrobots.txt is a plain text file placed in a website's root directory that tells web crawlers which pages can be crawled and which should be avoided
70%Verifiedrobots.txt is essentially an honor system — any crawler can bypass all restrictions simply by ignoring the file, and webmasters have no technical means to verify whether crawlers comply.
65%Verifiedrobots.txt is a plain text file placed in the root directory of a website that follows the Robots Exclusion Standard and is used to declare to crawlers which paths are allowed to be accessed.
65%VerifiedGooglebot determines the scope of crawlable pages by following each website's robots.txt protocol; robots.txt is a text file stored in the root directory of a website used to declare crawlable or blocked pages to web crawlers.
65%VerifiedThe robots.txt protocol is a gentleman's agreement and has no technically binding enforcement
65%Verifiedrobots.txt is an industry convention established in 1994 by the Robots Exclusion Protocol (REP) and is not a mandatory legal regulation, but mainstream search engines such as Google and Baidu strictly adhere to it
65%VerifiedMany AI training data crawlers tend to ignore the robots.txt protocol and rapidly scrape entire sites at extremely high concurrency, fundamentally differing from traditional search engine crawlers.
65%UnverifiedA growing number of content platforms have begun charging AI companies data licensing fees and introducing anti-scraping mechanisms.
60%UnverifiedMany mainstream AI training datasets were formed without strictly adhering to the robots.txt protocol.
60%Unverifiedllms.txt draws on the concept of robots.txt and aims to provide large language models with a clear and concise content index
60%Unverifiedrobots.txt is a 'gentleman's agreement' that relies on the crawler's voluntary compliance and has no enforcement power
60%Unverifiedllms.txt has a different positioning from robots.txt (which controls crawler access permissions) and sitemap.xml (which lists URLs); its goal is to let models quickly understand a site's structure and core content.
50%UnverifiedSome vendor documentation states that user-triggered AI agents may not follow robots.txt the way traditional crawlers do, because the requests are initiated by a real person.
50%UnverifiedAI training crawlers generally do not obey the robots.txt protocol, nor do they self-restrain their request frequency, with crawling behavior approaching a DDoS attack.
50%UnverifiedThere is no reciprocal relationship between the operators of AI training crawlers and websites, because they do not rely on search results for traffic, and therefore are not bound by the industry tacit understanding behind robots.txt.
50%Unverifiedrobots.txt User-agent matching is independent for each entry: a crawler first looks for the rule group whose name exactly matches its own UA, and only falls back to the User-agent: * wildcard rule when there is no match.
50%Unverifiedrobots.txt only controls whether crawling is allowed, not whether indexing is allowed; a URL that is disallowed from crawling may still appear in search results if it is linked from other sites (usually showing only the link without a snippet).
50%