Verified75% confidenceFactExact time
The robots.txt protocol was proposed by internet pioneer Martijn Koster in 1994 and is a plain text file placed in the root directory of a website
5
Sources
75%
Confidence
Long-term
Relevance
7/8/2026
First Seen
Sources
Learning Python Web Scraping from Scratch: How It Works, the Full Workflow, and Legal Boundaries
bilibiliPython全套学习7/6/2026
Related Claims
Verifiedrobots.txt is an industry convention established in 1994 by the Robots Exclusion Protocol (REP) and is not a mandatory legal regulation, but mainstream search engines such as Google and Baidu strictly adhere to it72% similarPartially Verifiedrobots.txt is a plain text file placed in a website's root directory that tells web crawlers which pages can be crawled and which should be avoided69% similarVerifiedGooglebot determines the scope of crawlable pages by following each website's robots.txt protocol; robots.txt is a text file stored in the root directory of a website used to declare crawlable or blocked pages to web crawlers.67% similarUnverifiedrobots.txt 和 sitemap.xml 作为网站向机器暴露元信息的早期实践，可被视为 WebMCP 的精神前身65% similarUnverifiedThe llms.txt proposal that emerged in 2024 follows the conventions of robots.txt, providing project descriptions in plain-text Markdown format at a website's root directory to help LLMs build an understanding.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/267481API
curl https://kongchang.com/api/v1/knowledge/claims/267481MCP
get_claim(id=267481)