Verified65% confidenceFactExact time
robots.txt is a plain text file placed in the root directory of a website that follows the Robots Exclusion Standard and is used to declare to crawlers which paths are allowed to be accessed.
3
Sources
65%
Confidence
Long-term
Relevance
9/13/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedWebMCP被视为继robots.txt和sitemap.xml之后面向机器的第三类网站元信息标准70% similarVerifiedURDF (Unified Robot Description Format) is the standard XML format in the ROS ecosystem for describing the physical structure of robots.65% similarUnverifiedDokBot allows users to upload PDF and text files to automatically build an AI customer service chatbot capable of answering questions and embedding it into a website60% similarUnverifiedseo-agent检测到AI爬虫访问时，动态提供干净的Markdown格式内容而非原始HTML55% similarUnverifiedTraditional browser automation tools (Selenium, Playwright, Puppeteer) require parsing complex DOM structures, and the generated page descriptions contain large amounts of noise that is useless to AI.55% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/919243API
curl https://kongchang.com/api/v1/knowledge/claims/919243MCP
get_claim(id=919243)