sitemap
A list of URLs intended for search engines, helping crawlers discover a site's pages. It is usually provided in XML format. It addresses page discovery but does not describe page semantics or priority.
Core Facts
Timeline (last 90 days)
ClaudeBot made 128 requests to /sitemap-posts.xml during the measurement period, accounting for two-thirds of the 192 requests to that file.
To let AI read the website, the priorities in order are: ensure robots.txt does not block crawlers and that the host/firewall does not intercept them, maintain a complete sitemap and declare it in robots.txt, ensure each page's pure HTML does not depend on script loading, and treat llms.txt as a courtesy to users and tools rather than a step toward gaining exposure.
A sitemap is a list of URLs for search engines, usually provided in XML format; it solves page discovery but does not describe the purpose or priority of the links.
All Facts (3)
A sitemap is a list of URLs for search engines, usually provided in XML format; it solves page discovery but does not describe the purpose or priority of the links.
65%UnverifiedTo let AI read the website, the priorities in order are: ensure robots.txt does not block crawlers and that the host/firewall does not intercept them, maintain a complete sitemap and declare it in robots.txt, ensure each page's pure HTML does not depend on script loading, and treat llms.txt as a courtesy to users and tools rather than a step toward gaining exposure.
50%UnverifiedClaudeBot made 128 requests to /sitemap-posts.xml during the measurement period, accounting for two-thirds of the 192 requests to that file.
50%