CCBot
The web crawler of the Common Crawl project, whose collected datasets are widely used for training large language models.
Core Facts
Timeline (last 90 days)
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
GPTBot, ClaudeBot, and Common Crawl are known AI training crawlers.
Emerging AI crawlers such as GPTBot, CCBot, and Amazonbot have flooded the web to collect training data in bulk, and some crawlers do not strictly comply with robots.txt restrictions or bypass filtering rules by disguising their User-Agent.
The access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
CCBot is the crawler for the Common Crawl project
All Facts (6)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%Partially VerifiedCCBot is the crawler for the Common Crawl project
65%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedGPTBot, ClaudeBot, and Common Crawl are known AI training crawlers.
50%UnverifiedEmerging AI crawlers such as GPTBot, CCBot, and Amazonbot have flooded the web to collect training data in bulk, and some crawlers do not strictly comply with robots.txt restrictions or bypass filtering rules by disguising their User-Agent.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%