The Tragedy of the Commons in the AI Era: Who Is Depleting the Internet's Public Resources

When the "Tragedy of the Commons" Meets Artificial Intelligence

In economics, there's a classic concept called the Tragedy of the Commons: when a public resource is available to everyone but no one is responsible for maintaining it, every rational individual tends to maximize their own benefit, ultimately leading to over-consumption or even complete depletion of that resource. This metaphor, originally about overgrazing on shared pastureland, is now replaying in a new form across the internet's open ecosystem—only this time, the "pasture" is the entirety of internet content, and the "overgrazing" is being done by AI crawlers and large language models.

The concept was formally introduced by ecologist Garrett Hardin in his 1968 paper of the same name, published in Science. Its core game-theoretic structure is clear and brutal: each herder gains +1 from adding one more cow, but the cost of grassland degradation is shared among everyone. Therefore, the marginal benefit of individual rational decisions always exceeds the marginal cost—until the commons collapses entirely. Notably, Nobel laureate Elinor Ostrom later provided an important correction through empirical research—she found that communities can effectively manage public resources through self-organized institutional arrangements, without necessarily relying on the two extreme solutions of privatization or government regulation. This insight offers a valuable third path for understanding content governance in the AI era.

A recent article titled The tragedy of the commons, AI edition, which sparked heated discussion on Hacker News, has thrust this age-old proposition back into the spotlight. It poses a sharp question: When AI companies indiscriminately scrape, learn from, and commercialize public content on the internet, are the "commons" that once sustained the internet's prosperity being quietly depleted?

hackernews source: The tragedy of the commons, AI edition

Who Is Consuming the Internet's Public Resources

The Creator's Dilemma: The Traffic Loop Is Being Severed

The internet has long operated on an implicit social contract: creators publish content and receive returns through advertising, subscriptions, or influence; search engines crawl content and direct traffic back to creators. This was a roughly balanced exchange.

In internet economics, this implicit contract is known as the foundational architecture of the "Attention Economy." The complete business logic chain works as follows: creators produce content → search engines index and rank it → users discover content through search → they click through to the original website → the site earns revenue through ad impressions (CPM/CPC models) or subscription conversions → revenue incentivizes more content production. Google sends trillions of clicks annually into this ecosystem, supporting a vast creative economy ranging from personal blogs to professional media.

However, the emergence of generative AI has broken this balance. Large models scrape massive quantities of web pages, forums, code repositories, and documentation for training, yet return virtually no traffic or revenue to the original creators. When users get answers directly from ChatGPT or other AI assistants, they no longer need to click through to the original website—this is the so-called "Zero-Click Answer" model. According to data from research firm SparkToro, even before AI's intervention, roughly 65% of Google searches ended with zero clicks (users got their answers directly on the search results page). The introduction of AI overview features has further intensified this trend, severing the already fragile connection between creators and readers even more.

Put simply: Creators put in the labor, AI companies extract the value, and the traffic loop that once sustained content creation has been cut off at the waist.

The Hidden Costs of Servers and Bandwidth: Who's Footing the Bill

Another layer of the tragedy of the commons manifests in technical costs. In recent years, a large number of open-source project maintainers and operators of small-to-medium websites have reported that traffic from AI crawlers is surging dramatically. These crawlers often scrape entire sites at high frequency and intensity, placing heavy bandwidth and compute pressure on servers that already depend on donations or personal maintenance.

Here's a concrete example: an open-source documentation site maintained by volunteers could see its server costs double overnight from an influx of AI crawlers, or even experience service instability. The party consuming resources (AI companies) sits on massive capital reserves, while the cost is borne by the contributors of those resources.

This is the most quintessential feature of the tragedy of the commons—privatizing the gains while socializing the costs.

Why This Is a Systemic Risk

Incentive Structures Are Collapsing

The most frightening aspect of the tragedy of the commons isn't the short-term consumption of resources—it's the long-term collapse of incentives. If creators discover that their painstakingly produced content is merely feeding AI models for free with no return whatsoever, the rational choice is to reduce output or simply lock content behind paywalls and closed platforms.

This creates a deep paradox:

AI models' capabilities are built precisely on open, high-quality public content.
Once this content dries up due to lack of incentives, the "nourishment" AI depends on will diminish as well.
Over-extraction ultimately harms the extractors themselves.

Everyone acts rationally in the short term, yet collectively marches toward an irrational outcome in the long term—this is the core logic of the tragedy of the commons.

The Spiral Decline of Data Quality and Model Collapse

A deeper concern is the "pollution" of the content ecosystem. As AI-generated content floods the internet, future models will inevitably digest increasing amounts of AI-generated data during training, creating the risk of what's known as Model Collapse.

The concept of model collapse was systematically articulated by research teams from Oxford and Cambridge in their 2023 paper The Curse of Recursion. The core mechanism is this: when AI models are trained on corpora containing large amounts of AI-generated data, the probability distribution of model outputs narrows with each generation. Rare but valuable information—the "long tail" of the distribution—is progressively forgotten, and outputs ultimately trend toward homogenization and degradation. Researchers liken this process to "inbreeding": each generation of model amplifies the biases of the previous one. Experiments show that after just 5-10 generations of recursive training, the diversity and accuracy of model-generated text show significant decline.

When the proportion of high-quality, human-created original content in the public content pool continues to decrease, the quality of the entire information ecosystem degrades in a spiral. This means that if original human content on the internet continues to shrink, the quality of future models' training data will enter an irreversible cycle of degradation. This isn't just a loss for content creators—it's a predicament faced by everyone who depends on internet information.

Possible Solutions

From Technical Defenses to Protocol Standards

Facing the tragedy of the commons, economics traditionally offers two solutions: privatization of property rights, and collective governance. In the contest between AI and content, both paths are being explored:

Technical defense: An increasing number of websites are actively protecting themselves by updating robots.txt, deploying anti-crawling mechanisms, and requiring AI crawlers to identify themselves. robots.txt is a plain text file placed in a website's root directory that follows the "Robots Exclusion Protocol," originally proposed by Martijn Koster in 1994. It uses User-agent and Disallow directives to tell web crawlers which pages can be crawled and which should be avoided. However, robots.txt is essentially a "gentleman's agreement"—it has no enforcement power and relies entirely on the crawler's voluntary compliance. In recent years, many AI companies' crawlers have been found ignoring robots.txt restrictions or using unidentified crawlers to circumvent rules. This has pushed site administrators toward more aggressive technical measures, such as behavior-based crawler detection, rate limiting, and JavaScript rendering challenges.
Commercial agreements: Some content platforms have chosen to sign data licensing agreements with AI companies, converting content crawling into paid licenses—essentially privatizing and pricing public resources. Between 2023 and 2024, several landmark deals emerged: Reddit signed a data licensing agreement with Google worth approximately $60 million/year; the Associated Press reached a news content licensing partnership with OpenAI; and Shutterstock established paid licensing frameworks for image data with multiple AI companies. However, this model has also sparked widespread controversy: large platforms have the bargaining power to negotiate favorable agreements, but individual creators on those platforms (the actual content producers) often cannot directly benefit from these deals. Meanwhile, institutions like The New York Times have chosen a different path—directly suing OpenAI and Microsoft for copyright infringement, attempting to establish the legal boundaries of AI training through judicial means. The coexistence of these different strategies reflects that the industry has yet to reach consensus on content value distribution in the AI era.

Building New Value Distribution Mechanisms

A more fundamental solution may lie in rebuilding mechanisms that allow value to flow back:

AI companies paying fair fees to content sources
Establishing industry-wide content licensing and copyright standards
Exploring new traffic and revenue-sharing models

The core objective of these approaches is the same: restoring the exchange contract that once sustained the internet's prosperity. Only when contributors can receive reasonable returns from the use of their content will the commons not be exhausted.

Returning to Ostrom's theoretical framework, successful governance of common resources typically requires several conditions: clearly defined boundaries, rules adapted to local conditions, collective decision-making mechanisms, effective monitoring, and graduated sanctions against violators. The current AI content ecosystem is precisely missing all of these dimensions—who has the right to use what data, where the boundaries of use lie, and who monitors enforcement all remain without clear answers.

This Isn't Just an AI Company Problem

The tragedy of the commons in the AI era reminds us that the explosive development of artificial intelligence didn't emerge from thin air—it's built on decades of public knowledge freely contributed by countless people. How to enjoy the dividends of AI without depleting and destroying the soil that nourished it is a question the entire industry, and indeed society, must confront.

Although this discussion only made limited ripples on Hacker News, it touches on a structural contradiction long ignored behind AI's boom:

The sustainability of technology ultimately depends on the sustainability of the ecosystem; and the sustainability of the ecosystem requires restraint and responsibility from every participant.