robots.txt Allows but CDN Denies: Real-World Test of AI Crawler Access Across a Thousand Sites

Most articles discussing AI crawler blocking focus only on robots.txt. But that file is just a request, not a lock—what actually determines what crawlers get is the combined action of the server, CDN, and bot management layer. A research organization conducted a real-world test on the top 1000 Tranco sites, and the results are worth revisiting for anyone who depends on AI search traffic.

Core Finding: 14% of Sites Are Fighting Their Own Policy

Among 549 sites that returned 2xx to a normal browser, 78 (14.2%) rejected at least one of GPTBot, OAI-SearchBot, ClaudeBot, or PerplexityBot—even though their robots.txt explicitly allows these crawlers. Excluding the 14 sites that were only rejected due to 429 rate limiting, there are still 64, accounting for 11.7%.

The significance of this number is that there's a crack between robots.txt and actual allowed behavior. For sites relying on AI search engine traffic, this crack means they may be quietly locking themselves out of the citation pools of ChatGPT Search, Perplexity, and other platforms.

CDN and WAF Layers Are the Real Gatekeepers

The most notable data comes from OAI-SearchBot. It determines whether ChatGPT Search can cite your content, yet it was rejected by servers on 8% of sites that explicitly allow it. These sites may have simply purchased some bot management rule, or flipped a "block AI crawlers" switch, inadvertently blocking search and answer crawlers as well.

Specific rejection rates by crawler:

• GPTBot: 14.4% rejected by servers, of which 11.1% occurred despite robots.txt allowing it
• ClaudeBot: 14.8% rejected, 10.9% despite being allowed
• PerplexityBot: 11.3% rejected, 8.4% despite being allowed
• OAI-SearchBot: 9.8% rejected, 8.0% despite being allowed

429 rate limiting is listed separately because it's a boundary case—"slow down" and "don't come back" are two different things. But when a crawler is rate-limited on its first request to the homepage from a single IP, the practical effect is equivalent to blocking.

The mirror situation also exists: 53 sites prohibit GPTBot in robots.txt but continue serving it normally. robots.txt is advisory in both directions.

The division of labor between CDN, WAF, and bot management layer. A CDN (Content Delivery Network) caches and distributes content nearby, a WAF (Web Application Firewall) intercepts malicious traffic, and the bot management layer is a policy layer deployed on top of both, specifically identifying and handling automated traffic. Major providers like Cloudflare, Akamai, and Fastly all have built-in bot scoring mechanisms based on user agent, IP reputation, and request fingerprints, and many offer one-click "block AI crawlers" switches. The problem is that these rules are often categorized by vendor rather than by purpose. A single "Block AI bots" switch may simultaneously block training crawlers and search answer crawlers. Blocking the former has little impact, but blocking the latter directly cuts off citation traffic from ChatGPT Search and Perplexity. This is why robots.txt says Allow but the server returns 403—the request never reaches the application layer; it's already blocked at the edge node.

Statistics on robots.txt Itself

From parsing the robots.txt of reachable sites:

• 82.7% of reachable sites have a robots.txt
• 25.1% block at least one search or answer crawler, 10.2% block most
• 26.0% block at least one training crawler, but only 2.6% do what the guidelines recommend: block only training crawlers while keeping search and answer crawlers open
• 2.5% have Content-Signal directives, 62.4% list a sitemap

The 2.6% figure itself is an insight. Most sites doing AI crawler management aren't making fine distinctions; they're handling all AI-related traffic with a blanket approach. The few sites that do distinguish correctly prove that categorizing by purpose rather than vendor is feasible.

The Real Adoption Rate of llms.txt Is Overestimated

Widely cited llms.txt adoption data is mostly polluted by "soft 404s"—sites return 200 for any path, but the content is an HTML "not found" page. The researchers re-crawled all reported llms.txt files, keeping only those whose content is a text file starting with a Markdown heading.

File Type	Any 2xx Response	Real File
llms.txt	34.3%	15.1%
llms-full.txt	21.9%	Not re-verified
agents.json	0.3%	—
Markdown version pages	3.5% provide, 1.5% link	—
WebMCP form annotations	Only 1 out of 254 sites with forms	—

The real llms.txt adoption rate is about one-seventh, not one-third. If you're building any tool that reads these files, test for soft 404s first: check the Content-Type, check if the first line is a heading.

Why soft 404s pollute statistics. A soft 404 is when a server returns HTTP 200 for a non-existent path, but the actual content is an HTML "page not found" message. Many single-page applications, static hosting platforms, and CDN default configurations handle it this way—as long as the path matches some catch-all route, it always returns 200. If a crawler scanner only looks at status codes, it will count llms.txt, agents.json, and ai.txt under every domain as "present," inflating adoption rates. The way to identify soft 404s is to check whether the Content-Type is text/plain (or text/markdown), and whether the first line of the response body starts with a Markdown heading (#). After re-crawling accordingly, the real llms.txt adoption rate dropped from 34.3% to 15.1%, indicating that nearly 60% of "present" were false positives. Any adoption survey based on HTTP status codes should first filter for soft 404s.

Overall Readiness Score

Using the research organization's checker score (v2, which only counts crawlers that fetch pages for AI search and answers, and treats active blocking as a legitimate choice rather than a failure): mean 84.5, median 90. 355 sites are AI-ready, 106 are actively restricted, 88 are JavaScript-dependent (crawlers are allowed in, but content is rendered client-side and most crawlers don't execute JS), and 126 could only be partially detected because they blocked the checker itself.

The 88 JavaScript-dependent sites are another hidden risk group. They are open to AI crawlers at both the robots.txt and server levels, yet their content cannot be crawled due to how it's rendered.

Why JavaScript dependence constitutes a hidden risk. Modern front-end frameworks (React, Vue, Next.js client-side rendering mode) often inject content into the DOM only after executing JavaScript in the browser. Traditional crawlers only fetch the initial HTML returned by the server and don't execute JS, so they see an empty shell page. Mainstream AI crawlers like GPTBot, ClaudeBot, and PerplexityBot currently generally don't execute JavaScript, meaning that even if both robots.txt and WAF allow them through, the content they get is still empty. The solution is server-side rendering (SSR), static site generation (SSG), or prerendering, so that content exists in the initial HTML. The 88 sites in the study with affected "AI-ready" scores fall into this category—they are completely open at the policy level but uncrawlable at the technical level, which is more insidious than explicit blocking because site owners often assume they are open.

How to Test Your Own Site in One Minute

The fastest method is to run two commands from the terminal. If the first returns 200 and the second returns 403, then your robots.txt isn't the whole story:

curl -sI -A "Mozilla/5.0" https://example.com/ | head -1
curl -sI -A "OAI-SearchBot/1.0; +https://openai.com/searchbot" https://example.com/ | head -1

The comparison script used in the study is a dependency-free Node file that can point to any domain CSV. The full readiness check is available at crawlreadiness.com, running the exact code used in the study, with reference pages for each of the 51 crawlers, including documented user agents, purposes, whether they respect robots.txt, and the exact robots.txt lines that allow or block them.

Why use real user agents for request verification. The -A parameter of curl sets the User-Agent request header, and the server uses this string to determine whether the visitor is a normal browser or a specific crawler. The first command masquerades as a Mozilla browser, and the second forges the official OAI-SearchBot identifier. Comparing the status codes returned by both tells you whether the server behavior is consistent with robots.txt: if the browser gets 200 and the crawler gets 403, a CDN or WAF layer rule is at work; if both get 200, it's allowed; if both get 403, the application layer is actively blocking. Advanced verification should also check whether the returned content is an empty shell (JS-dependent) or whether it's redirected to a verification page. Note that user agents can be forged, so this only verifies "how the server responds to your claimed identity," not authentication. Before doing destructive testing, it's advisable to check the crawler's official documentation to confirm its real user agent format.

If You Fall Into That 14%, What Should You Do

If you find yourself affected, the path to resolution is clear:

1. Look at the bot management layer first, then robots.txt. On Cloudflare, it's Security → Bots; "Block AI bots" will block both search and answer crawlers. On other CDNs and WAFs, look for managed rules matching the above user agent strings.
2. Decide by purpose, not by vendor. Blocking GPTBot (training) while allowing OAI-SearchBot (ChatGPT search citations) is a reasonable and common choice, and the 2.6% of top sites that seriously do this prove it works.
3. Verify through requests, not by reading files. The curl pair above takes only ten seconds.

The data is released under CC BY 4.0, including per-site CSV, per-crawler CSV, and summary JSON, re-run monthly with history retained. If you find misclassified sites, the per-site rows show every status code received.

One-line Summary

14% of top sites quietly block AI crawlers at the CDN/server layer while robots.txt allows them; real llms.txt adoption is only 15%.

Paragraph Summary

A real-world test of the top 1000 Tranco sites reveals a significant crack between robots.txt and actual allowed behavior. Among 549 normally reachable sites, 14.2% rejected at least one AI crawler that robots.txt explicitly allows, and even after excluding 429 rate limiting, 11.7% remained. OAI-SearchBot determines whether ChatGPT Search can cite content, yet it was rejected by servers on 8% of sites that allow it, often due to bot management rules from CDN or WAF causing collateral damage. The reverse also exists: 53 sites that prohibit GPTBot in robots.txt continue to serve it. The real adoption rate of llms.txt is about 15.1%, far below the widely cited 34.3%, with most statistics polluted by soft 404s. Only 2.6% of sites achieve the fine-grained strategy recommended by guidelines—blocking only training crawlers while keeping search and answer crawlers open. 88 sites have content that cannot be crawled due to JavaScript rendering, constituting a hidden risk.