Unverified50% confidenceFactExact time
robots.txt is an access-control mechanism that tells crawlers what can and cannot be crawled; llms.txt is a content navigation tool that proactively recommends core content worth reading to models
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
Partially Verifiedrobots.txt is a plain text file placed in a website's root directory that tells web crawlers which pages can be crawled and which should be avoided78% similarVerifiedGooglebot determines the scope of crawlable pages by following each website's robots.txt protocol; robots.txt is a text file stored in the root directory of a website used to declare crawlable or blocked pages to web crawlers.76% similarUnverifiedThe llms.txt proposal that emerged in 2024 follows the conventions of robots.txt, providing project descriptions in plain-text Markdown format at a website's root directory to help LLMs build an understanding.72% similarUnverifiedllms.txt is an emerging website standard proposal, similar to robots.txt but specifically targeting AI crawlers, written in Markdown format.72% similarUnverifiedrobots.txt 和 sitemap.xml 作为网站向机器暴露元信息的早期实践，可被视为 WebMCP 的精神前身66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/895256API
curl https://kongchang.com/api/v1/knowledge/claims/895256MCP
get_claim(id=895256)