llms.txt in Practice: Zero Crawls in 30 Days — What Are AI Crawlers Actually Reading?

llms.txt was once hailed as a breakthrough — a way to help AI tools quickly understand a website. But 30 days of real-world data from one developer poured cold water on that hope: AI crawlers identified by Cloudflare made 0 requests to /llms.txt, while /robots.txt received 605 requests over the same period. This result doesn't invalidate llms.txt's reason for existing, but it forces us to rethink: if you want AI to read your site, where should you actually invest your effort?

30-Day Test: Zero Crawls — But Is the Data Credible?

From September 14 to October 13, 2026, blogger Yap Pat Yih used Cloudflare's AI Crawl Control (available on the free plan) to track AI crawler access paths. After filtering for all paths containing llms, the dashboard returned "No requests data found"; a follow-up check the next day using 7-day and 30-day windows still showed zero.

Zero itself needs verification. The author ran three cross-checks: the same path filter found 97 requests to /sitemap-posts.xml within 7 days; filtering by .txt suffix found records of BingBot fetching IndexNow key files; and the llms files and article pages are generated in real time by the same Worker — while article pages were requested 57 times in the same period. The filter works, text files are counted, and the generation path is covered — the zero is real.

Cloudflare AI Crawl Control is a feature Cloudflare launched in 2025 that lets site administrators see which AI crawlers have visited their site, which paths they've crawled, and optionally block or charge them. It's based on Cloudflare's edge network logs, identifying crawlers via UA strings, known IP ranges, and behavioral characteristics. However, identification itself depends on the crawler directory maintained by Cloudflare, so newly emerged crawlers that aren't listed, or tools masquerading as browsers, won't be counted. This is also why the article's "zero crawls" conclusion must be qualified to the scope of "crawlers identified by Cloudflare."

What Crawlers Are Actually Reading: robots.txt, sitemap, and the Pages Themselves

During the same period, AI crawlers identified by Cloudflare made roughly 3,000 requests to the site. The distribution is telling:

• /robots.txt: 605 requests
• Homepage /: 207 requests (Googlebot 39, GPTBot 29, OAI-SearchBot 25, BingBot 20, ClaudeBot 11)
• /sitemap-posts.xml: 192 requests (ClaudeBot alone accounted for 128, two-thirds of the total)
• Individual articles: 57 requests (PerplexityBot, GPTBot, CCBot, Claude-User, and ClaudeBot all appeared)
• /llms.txt and /llms-full.txt: 0 requests

In the site-wide crawler request ranking, Googlebot (602), ClaudeBot (464), GPTBot (356), Applebot (221), and Meta-ExternalAgent (185) topped the list. These crawlers come frequently, but they read only the "usual suspects": robots.txt for rules, sitemap for pages, and the pages themselves for content. llms.txt is completely absent from their access list.

robots.txt is a plain-text protocol file placed in a site's root directory, using directives like User-agent and Disallow to tell crawlers which paths they may and may not crawl. It isn't mandatory per se, but mainstream search engines and AI companies treat it as a de facto standard. sitemap (usually in XML format) lists the URLs of pages the site wants indexed along with their update times, helping crawlers discover deep pages and avoid missing them. The two serve different purposes: robots.txt governs "can you crawl it," while sitemap governs "what pages are there to crawl." In this article, ClaudeBot directed two-thirds of its requests to the sitemap, showing that sitemaps are the primary entry point for AI crawlers to discover content.

What the Proposers and AI Companies Say

Jeremy Howard proposed llms.txt on llmstxt.org on September 3, 2024, but the proposal itself didn't put crawlers first. It describes an "on-demand" file — one that AI tools read only when they need to understand a topic while assisting a user. The proposal's original wording: "Our expectation is that llms.txt will primarily be used for inference rather than training." The author's measurements only cover crawlers, so they're actually testing the use case the proposal doesn't emphasize.

Google's Search Central documentation explicitly states that appearing in Google's AI features does not require creating new machine-readable files or AI text files, and makes no mention of llms.txt throughout. OpenAI, Anthropic, and Perplexity each provide documentation for their own crawlers, all pointing to robots.txt as the control entry point; none declare that their crawlers read a site's llms.txt. Interestingly, all three companies publish llms.txt for their own developer documentation — which aligns perfectly with the proposal's intent: tools read it when asked to read documentation, rather than crawlers sweeping the entire web.

GPTBot, ClaudeBot, and PerplexityBot are the crawler identifiers used by OpenAI, Anthropic, and Perplexity, respectively, to scrape web content. Each publishes documentation explaining its crawler's UA string, purpose, and how sites can opt out — but the opt-out method consistently points to robots.txt. In contrast, OAI-SearchBot is OpenAI's crawler for search indexing, and Claude-User is used to fetch pages in response to users' real-time requests; the two serve different purposes from training crawlers. Understanding these crawlers' division of labor helps explain why the real entry point for "getting AI to read your site" remains robots.txt and the pages themselves.

What This Measurement Can't See

The data has its limits, and the author honestly acknowledges them. Cloudflare only counts crawlers it can "identify," so three types of access are invisible: people manually opening /llms.txt in a browser; crawlers Cloudflare doesn't recognize; and AI tools masquerading as ordinary browsers that fetch the file while assisting users. That last category is precisely the core scenario the proposal describes, and it cannot be ruled out.

The time window is also limited: the dashboard only offers a maximum 30-day range, other analytics on the free plan can't be filtered by path, and the Worker doesn't retain logs. So the conclusion that can be drawn is narrow — during these 30 days, AI crawlers identified by Cloudflare never requested this file.

So Should You Keep llms.txt?

The author's choice is to keep it, for two reasons. Zero cost: the Worker generates both files in real time from the site's existing articles, requiring no manual maintenance. User-friendly: when someone or an AI tool is pointed to the site, there's a clean plain-text file to read instead of a layout-heavy page.

"If no one comes asking for it, I've lost nothing. But I won't treat it as a path to being discovered." That sentence is worth remembering for every content site owner.

What to Do If You Want AI to Read Your Site

The measured data gives a clear priority order:

1. Make sure crawlers can get in. Check that robots.txt isn't blocking them and that your host or firewall isn't intercepting them.
2. Maintain a complete sitemap and declare it in robots.txt. Two-thirds of ClaudeBot's requests went to the sitemap — you can't skip this step.
3. Make every page pure HTML, with text that renders without relying on scripts.
4. If you add llms.txt, treat it as a courtesy to readers and tools, not a step toward visibility — and measure whether anyone actually requests it.

FAQ

What is llms.txt?

A Markdown file located at the site root /llms.txt, containing the site name, a one-sentence introductory summary, and a list of main pages with brief descriptions. Jeremy Howard proposed it on llmstxt.org on September 3, 2024, to help AI tools quickly understand a website when users ask about it.

Does Google use llms.txt?

Google Search Central's page on AI features states that no new machine-readable files or AI text files need to be created to appear in Google's AI features, and it doesn't name llms.txt. Adding it is not a step toward Google AI answers.

Do AI crawlers read llms.txt?

In this site's real-world test, no. Within 30 days, AI crawlers identified by Cloudflare made zero requests to /llms.txt and /llms-full.txt, while robots.txt was requested 605 times in the same period. The crawler documentation from OpenAI, Anthropic, and Perplexity all points to robots.txt and makes no mention of llms.txt.

Should I add llms.txt to my site?

If you can keep it accurate at low cost, yes — as a courtesy to users and tools looking for it. But don't expect it to bring crawler visits or search traffic — the test showed zero over 30 days. Prioritize robots.txt, sitemap, and pure HTML pages.

What's the difference between llms.txt and robots.txt?

robots.txt tells crawlers which paths they may crawl; OpenAI, Anthropic, and Perplexity all document it as the way to control crawlers. llms.txt is a site summary and page list for AI tools to read when users ask about the site. In the test, AI crawlers fetched robots.txt 605 times in 30 days and llms.txt zero times.