Only 8.9% of Websites Block AI Crawlers, Yet 94.8% Have Never Been Cited in AI Answers

A Data Paradox Hiding in Plain Sight

As generative AI becomes an increasingly important gateway for information access, the relationship between websites and AI crawlers is undergoing a subtle but profound transformation. Recent research data reveals a fascinating phenomenon: only 8.9% of websites choose to block AI crawlers, yet a staggering 94.8% have never been cited in AI-generated answers.

Placed side by side, these two numbers paint a true picture of the current AI content ecosystem — the vast majority of websites remain open to AI crawling, yet very few ever gain exposure in AI-generated responses. This isn't merely a technical statistic; it reflects a fundamental reshuffling of content distribution power in the AI era.

hackernews source: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

Why Most Websites Don't Block AI Crawlers

Openness Is the Default State

The 8.9% blocking rate might seem low, but it actually reflects a basic inertia of the internet: openness is the default state, and blocking requires deliberate action. Most website operators are either unaware of AI crawlers' existence or lack a clear strategy for dealing with them.

Blocking AI crawlers like GPTBot, ClaudeBot, and CCBot via robots.txt requires website administrators to actively configure their settings. robots.txt is a plain text file placed in a website's root directory that tells web crawlers which pages can be crawled and which should be avoided. This protocol was first proposed by Dutch engineer Martijn Koster in 1994 and has since become a foundational convention of the internet. However, robots.txt is essentially a "gentleman's agreement" — it relies on the crawler's voluntary compliance and has no enforcement power. GPTBot is OpenAI's crawler identifier made public in August 2023, ClaudeBot belongs to Anthropic, and CCBot is the crawler for the Common Crawl project. Website administrators need to add Disallow rules for each of these User-Agents in their robots.txt, which requires a certain level of technical competence.

For the vast number of small websites and personal blogs, they often lack both the technical capability and the motivation to do this. In other words, a significant portion of the 91.1% that don't block are "passively open" rather than "actively welcoming."

Blockers Are Mostly Major Media Outlets

Interestingly, the websites that choose to block AI crawlers tend to be major media outlets and publishing institutions with high-value content. Large media organizations like The New York Times and Reuters were among the first to block AI crawlers in their robots.txt files, and some have even pursued legal action to assert their content rights. This means that while the 8.9% of blockers are few in number, they likely far exceed the average in terms of content quality and influence.

What the 94.8% Citation Gap Really Means

AI Answer Exposure Is Highly Concentrated Among a Few Sources

If blocking data reflects websites' "defensive posture," then the 94.8% never-cited figure reveals the "selection bias" of AI answers. The overwhelming majority of websites, even when completely open to AI crawling, cannot secure a place in final AI-generated responses.

This concentration phenomenon is closely related to the prevailing Retrieval-Augmented Generation (RAG) architecture. RAG works as follows: when a user asks a question, the system first retrieves the most relevant document fragments from a pre-built knowledge index, then feeds these fragments as context into a large language model, which synthesizes them into a final answer. In this architecture, "most relevant" is typically determined by weighted ranking across multiple signals including semantic similarity, source authority, and content freshness. This means high-authority information sources naturally dominate retrieval rankings, while long-tail content — even if indexed — rarely wins the competition to enter the final context window.

Wikipedia, major news organizations, and professional knowledge platforms capture the vast majority of citation share. For long-tail websites, opening up to crawling brings no visible traffic return — they silently contribute training data without receiving any attribution or traffic.

The Collapse of Traditional SEO Long-Tail Traffic Logic

In the traditional search engine era, even lower-ranked websites still had a chance to receive trickle traffic through second and third page results. But in the AI answer ecosystem, this "long-tail exposure" has virtually disappeared. AI typically synthesizes citations from only a handful of sources, forming a closed answer that eliminates the need for users to click through to original websites.

This poses a severe challenge for content creators who depend on search traffic: content is used, but authors remain invisible, and websites go unvisited. This is also a recurring anxiety point in HackerNews community discussions.

The Content Creator's Dilemma with AI Crawlers

The Open-or-Block Decision

This data puts a pointed question before all content creators: when it comes to AI crawlers, should you open up or block?

Choose openness: Your content may be used to train AI models, but the probability of gaining citation exposure is less than 5.2%, with highly uncertain returns.
Choose blocking: You can protect your content from being used without compensation, but you may completely forfeit any chance of being cited in the AI answer ecosystem — and AI search is becoming the preferred entry point for an increasing number of users.

This is effectively a prisoner's dilemma. The prisoner's dilemma is a classic game theory model describing how rational individuals, lacking coordination mechanisms, may make choices that are collectively detrimental. In the AI crawler context, if all small and medium websites collectively blocked AI crawlers, they might force AI companies to seek paid licensing; but a single website's block has virtually no impact on the AI training data pool, and may instead cause it to miss the slim chance of being cited. This "individual rationality leading to collective irrationality" structure makes it difficult for small creators to change the game through individual action. Only external coordinating forces — industry associations, technical standards bodies, or policy regulation — can potentially break the deadlock.

A Severe Imbalance in Value Distribution

The deeper issue lies in the imbalance of value distribution. Content creators invest time and resources to produce original content, AI companies crawl this content to train models and generate answers, yet ultimately it's the AI platforms that capture the vast majority of user attention and commercial value. The original content contributors occupy the most disadvantaged position in this chain.

This also explains why major media outlets are increasingly choosing legal confrontation or commercial partnerships (such as OpenAI's content licensing agreements with multiple publishing institutions) — they have bargaining power, while long-tail creators can only passively absorb the consequences. As of 2024, OpenAI has signed content licensing agreements with the Associated Press, Axel Springer (which owns Politico and Business Insider), Le Monde, Prisa Media, and other media organizations, with licensing amounts reportedly ranging from several million to tens of millions of dollars. Google has also reached agreements with some publishers through its "Extended Crawling" policy and separate deals. However, these transactions only benefit large media groups with bargaining power. For small and medium creators who make up the vast majority of internet content by volume, no viable large-scale compensation mechanism currently exists.

Future Directions for Breaking the AI Crawler Impasse

Citation Attribution Mechanisms May Be the Key Breakthrough

To break this impasse, AI platforms' citation and attribution mechanisms will be critical. Products like Perplexity that explicitly label sources and provide clickable links are, to some extent, attempting to rebuild a traffic bridge between AI answers and original content. Perplexity AI is an AI search product that markets itself on citation transparency, explicitly labeling the source URL for each piece of information when generating answers and providing clickable redirect links. This contrasts with Google's AI Overviews — which occasionally provide source links but, because the answer is already fully presented on the search results page, significantly reduce users' willingness to click. Research shows that Google AI Overviews have reduced organic search result click-through rates by approximately 30-40%. While Perplexity's model is more "friendly," its actual traffic-driving effect remains limited, because the completeness of AI answers inherently diminishes users' motivation to visit original sources.

Only if AI answers can more fairly cite and drive traffic will websites' openness yield actual returns.

Restructuring the Content Ecosystem's Business Model

In the long run, the content ecosystem may need an entirely new business model. Whether it's paid content licensing, traffic-sharing from AI platforms, or some contribution-based compensation mechanism, a more balanced value distribution between AI companies and content creators needs to be established. Some emerging solutions such as content provenance protocols in the Web3 space and blockchain-based attribution systems are still in early experimental stages, with their feasibility and scalability yet to be proven.

The numbers 8.9% and 94.8% are essentially reminding us: in the process of AI reshaping information distribution, if the value distribution problem isn't solved, the sustainability of the content ecosystem will face a genuine threat. Between the open spirit of the internet and creators' legitimate rights, a new equilibrium still needs to be found.