AI Crawlers Out of Control: Over Half of kernel.org's Computing Power Devoured

When Crawlers Become a Server's Biggest Burden

Open source infrastructure is facing an unprecedented crisis: web crawlers built to harvest AI training data are draining public servers' computing resources in ways that border on malicious. Konstantin Ryabitsev, a core maintainer of the Linux kernel project, recently revealed the grim reality facing git.kernel.org in a blog post titled Creepy Crawlies.

As the official Git repository for the Linux kernel, git.kernel.org handles legitimate access from developers worldwide — code clones, commit browsing, and more. But the numbers Ryabitsev shared are staggering:

TL;DR: We spend more CPU rendering commit pages for crawlers than we do serving all other legitimate traffic combined, including git clones. At any given time, across 5 geographically distributed nodes, there are 14 CPU cores doing nothing but rendering git commit HTML pages for crawlers.

In other words, a critical piece of infrastructure serving the global open source community is spending the majority of its computing power not on real users, but on the relentless, background-radiation-like bombardment of AI crawlers.

rss source: Creepy crawlies

Why Git Repositories Are a Prime Target for AI Crawlers

A Nearly Infinite URL Space

Git repository web frontends — cgit, gitweb, and the like — are a natural nightmare for crawler management. Every commit has its own HTML page. Every historical version of every file can generate a URL. Diff views, blame views, tree-browsing by path — these combinations produce a nearly infinite crawlable page space.

For the Linux kernel repository, with hundreds of thousands of commits spanning over thirty years of history, the number of unique crawlable URLs is astronomical. What makes this especially punishing is that rendering these pages isn't a matter of serving static files. Each request triggers real-time Git command execution, diff computation, and syntax-highlighted HTML generation — every single hit translates directly into real CPU consumption.

AI Crawlers No Longer Play by the Rules

The traditional etiquette of web crawling — respecting robots.txt, throttling request rates, identifying with honest User-Agent strings — is breaking down in the current wave of AI data harvesting. To feed training data to large models, vast numbers of AI crawlers employ aggressive tactics: distributed IPs, spoofed identities, and outright disregard for crawl rate limits. They have no interest in the capacity of target servers; they only care about grabbing as much data as fast as possible.

This "cost-be-damned" approach shifts the computational burden entirely onto the public infrastructure being scraped — infrastructure that, in the case of open source projects running on volunteer labor and donations, amounts to outright resource extraction.

Beyond kernel.org: A Crisis Spreading Across Open Source Infrastructure

This post resonated so widely because it hits a raw nerve across the entire open source and open web ecosystem. Simon Willison, a prominent developer and creator of the Datasette project, admitted when sharing the article that it worries him deeply from a Datasette perspective — Datasette is an open source tool for exploring and publishing data that generates enormous numbers of crawlable pages.

I worry about this a lot from the Datasette perspective, given that it can expose a truly enormous number of crawlable pages.

Over the past year or more, organizations ranging from the Wikimedia Foundation to various code hosting platforms and documentation sites have been reporting surging traffic and rising costs driven by AI crawlers. Many smaller open source projects have been forced to add rate limiting, CAPTCHAs, or simply shut down parts of their public-facing functionality in response — a direct contradiction of the open-sharing ethos these projects were built on.

Technical Defenses and the Deeper Contradiction

The Limited Toolkit Available to Operators

Facing runaway AI crawlers, server operators have a fairly limited set of options:

Caching rendered output: Cache frequently accessed commit pages as static content to avoid redundant computation;
Rate limiting and IP blocking: Detect abnormal access patterns and throttle them, though effectiveness drops significantly against distributed crawlers;
Bot challenges: Deploy proof-of-work systems like Anubis that force crawlers to pay a computational cost, though this degrades the experience for legitimate users;
Restricting deep crawls: Use robots.txt and path rules to guide crawlers — assuming they bother to comply.

None of these are real solutions. They all involve trading away openness and accessibility. When a public resource has to build walls around itself just to survive, the original open ideal of the web has already been compromised.

Anubis deserves a closer look as a defensive tool. Created by open source developer Xe Iaso, its core idea is to introduce a proof-of-work challenge at the HTTP layer: when the server detects a suspected crawler request, it requires the client's browser to solve a hash puzzle locally (typically SHA-256-based) before granting access. A normal user's browser can solve this transparently in milliseconds with no perceptible delay; large-scale concurrent crawlers, however, face multiplied computational overhead that makes the operation economically unviable. The approach has clear limitations: headless browser crawlers can execute JavaScript and pass the challenge, and it can also block legitimate bots that rely on raw HTTP fetching, such as search engine crawlers. Anubis is therefore more of a mitigation that "raises the cost of attack" than a definitive solution.

The Ethical Reckoning Over AI Data Harvesting

At its core, this crisis is an AI ethics problem. As hinted by the ai-ethics tag on the original post, the root issue is this: AI companies harvest data to train models and generate enormous commercial value, while externalizing the cost of that harvesting onto the maintainers of public infrastructure.

The data being scraped — whether Linux kernel commit history or open source documentation — is public wealth created for free by countless contributors. Today, that public wealth is being harvested at scale to train closed, commercial models, while the people hosting it are left to foot the resulting compute bill. This imbalance of rights and responsibilities is a structural problem at the heart of today's AI data ecosystem that urgently needs to be confronted.

This issue is also legally contested. Some AI companies invoke "fair use" to justify large-scale crawling, but this doctrine varies in scope across legal systems and addresses copyright rather than the actual harm caused to servers by crawling activity. Mechanisms currently under discussion in academia and industry include: mandating that AI crawlers respect dedicated bot identifiers in robots.txt (such as GPTBot, CCBot, etc.), establishing data licensing markets, and creating collective licensing organizations modeled on broadcast copyright bodies, whereby AI companies pay content providers a "training data usage fee." However, none of these proposals have an enforceable international framework behind them, and a large number of overseas crawler operators exist in a regulatory vacuum.

The Open Web Stands at a Crossroads

The image of 14 CPU cores at kernel.org doing nothing but serving crawlers is a powerfully symbolic snapshot. It reminds us that beneath the AI wave, openly shared web infrastructure is under pressure unlike anything it has faced before. Without more reasonable crawling standards, data harvesting protocols, and meaningful accountability from AI companies for the public resources they consume, more and more open services may be forced to close themselves off.

For every developer and organization running a crawlable website, this is not a distant concern — it is a present reality. How to stay open while defending finite computing resources is a question that will be inescapable for a long time to come.