Common Crawl
A publicly accessible database that continuously crawls internet content. Operating since 2008, it has accumulated raw crawl data from over 25 billion web pages and is one of the largest public web corpora in the world.
Core Facts
Common Crawl is a nonprofit organization that has continuously crawled internet content since 2008 and makes its datasets freely available
English data accounts for an overwhelming proportion of mainstream large language model training corpora; taking public datasets such as Common Crawl as an example, English content typically constitutes 40%–60% of the total, while languages such as Vietnamese and Chinese often account for less than 5%.
CCBot is the crawler for the Common Crawl project
Timeline (last 90 days)
Common Crawl提供原始HTML数据，极度真实但脏，适合测试清洗和解析管道，不适合快速出成果。
Failed queries should be stored as 'error' rather than 'not included', and the indexing rate should only be calculated over URLs that actually received an answer, to prevent errors from being misread as the site being dropped.
On October 10 and 11, the script received a 521 directly when requesting the collection list; because that call had no retry mechanism, the entire phase failed without writing data, whereas URL queries had already had retries added since August.
The author acknowledges that the time-window comparison has selection bias, because most non-scheduled queries were manually re-run after seeing morning failures, and by the time re-runs occurred later in the day the upstream service may have already recovered.
During the 03:00–06:00 UTC window, 491 out of 1,339 queries failed, a failure rate of approximately 37%; in other time windows, 114 out of 550 queries failed, a failure rate of approximately 21%.
The CDX API (Capture Index API) is the standard interface for accessing Common Crawl index metadata; it can query whether a URL has been indexed, when it was indexed, and the HTTP status, but does not return the page content itself.
Among the 173 snapshots, every query in 11 snapshots failed; October was the worst month, with 79 failures out of 132 queries.
The distribution of failure types was 344 HTTP 503s, 103 HTTP 504s, 80 HTTP 521s, 52 timeout interruptions, 18 502s, 7 500s, and 1 520.
A developer uses a script running on a Cloudflare Worker to query the Common Crawl index once per day, checking whether approximately ten-odd URLs have been indexed.
Accumulating from May 15, 173 snapshots generated 1,889 queries, of which 605 received no answer at all, accounting for 32% of all queries.
12 more timeline events
All Facts (20)
Common Crawl is a nonprofit organization that has continuously crawled internet content since 2008 and makes its datasets freely available
70%VerifiedEnglish data accounts for an overwhelming proportion of mainstream large language model training corpora; taking public datasets such as Common Crawl as an example, English content typically constitutes 40%–60% of the total, while languages such as Vietnamese and Chinese often account for less than 5%.
65%Unverified整个互联网可抓取的公开网页数据（Common Crawl）约400TB
70%Partially VerifiedCCBot is the crawler for the Common Crawl project
65%UnverifiedCommon Crawl提供原始HTML数据，极度真实但脏，适合测试清洗和解析管道，不适合快速出成果。
50%UnverifiedAccumulating from May 15, 173 snapshots generated 1,889 queries, of which 605 received no answer at all, accounting for 32% of all queries.
50%UnverifiedFailed queries should be stored as 'error' rather than 'not included', and the indexing rate should only be calculated over URLs that actually received an answer, to prevent errors from being misread as the site being dropped.
50%UnverifiedThe author acknowledges that the time-window comparison has selection bias, because most non-scheduled queries were manually re-run after seeing morning failures, and by the time re-runs occurred later in the day the upstream service may have already recovered.
50%UnverifiedDuring the 03:00–06:00 UTC window, 491 out of 1,339 queries failed, a failure rate of approximately 37%; in other time windows, 114 out of 550 queries failed, a failure rate of approximately 21%.
50%UnverifiedThe CDX API (Capture Index API) is the standard interface for accessing Common Crawl index metadata; it can query whether a URL has been indexed, when it was indexed, and the HTTP status, but does not return the page content itself.
50%UnverifiedAmong the 173 snapshots, every query in 11 snapshots failed; October was the worst month, with 79 failures out of 132 queries.
50%UnverifiedThe distribution of failure types was 344 HTTP 503s, 103 HTTP 504s, 80 HTTP 521s, 52 timeout interruptions, 18 502s, 7 500s, and 1 520.
50%UnverifiedMotif relies heavily on NVIDIA's open-source Nemotron pretraining dataset, supplemented by internal data from public datasets and processed Common Crawl data that has undergone basic filtering and deduplication.
50%UnverifiedTraining datasets that rely primarily on web crawlers as their source (such as Common Crawl) contain large amounts of low-quality, duplicate, and even fabricated content.
50%UnverifiedGPTBot, ClaudeBot, and Common Crawl are known AI training crawlers.
50%UnverifiedIt is estimated that mainstream web crawl datasets such as Common Crawl had largely covered publicly accessible high-quality English web pages around 2020.
50%UnverifiedCommon Crawl is one of the most commonly used public datasets in AI training, serving as an important data source for the GPT series of models and numerous open-source large language models.
50%UnverifiedThe compliance of commonly used pre-training datasets such as Common Crawl and Books3 has been called into question
50%UnverifiedDomain name popularity scores can use Tranco rankings, Common Crawl crawl frequency, or DNS query volume as weights for offline sorting during the index-building phase.
50%UnverifiedLarge-scale image-text datasets (such as LAION-5B and Common Crawl-derived data) are primarily collected from publicly available internet resources and exhibit uneven distribution issues, including English content dominance and overrepresentation of Western aesthetic standards.
50%