GPTBot
A web crawler developed by OpenAI for bulk collection of publicly available web data to support AI model training. It identifies itself and respects site-level access restrictions declared through the robots.txt protocol.
Core Facts
Timeline (last 90 days)
The opt-out cost for the three types of AI crawlers forms a gradient: training crawlers < search crawlers < user-triggered fetchers
At present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
Blocking GPTBot in order to appear in ChatGPT depends on which OpenAI agent you mean; if search visibility matters, you should consult the latest user agent documentation before writing rules.
OpenAI specifically defines crawlers such as GPTBot to decide which content can be read, for tools such as ChatGPT search.
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
Testing AI crawler reachability by spoofing user agents locally has limitations: some CDNs verify bots by IP, so a fake GPTBot may be blocked while the real one is allowed through, and local blocking results are therefore only a strong hint that must be confirmed with CDN logs and origin logs.
Attackers heavily forge the User-Agents of AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, with a single IP rotating through User-Agents such as KimiBot, Qwenbot and Hunyuan within a single session while requesting the `.env` file.
OpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.
The user-agent is an HTTP request header field that clients use to self-report their identity, usually in a format such as 'Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)'.
Among the 4,242 robots.txt files parsed in the September 28 snapshot, GPTBot was rejected 1.60 times for every one time it was allowed; for OAI-SearchBot this ratio was only 0.91.
7 more timeline events
All Facts (18)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%Partially VerifiedGPTBot is OpenAI's crawler identifier made public in August 2023
65%UnverifiedMainstream AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google DeepMind), and Bytespider (ByteDance).
60%UnverifiedThe opt-out cost for the three types of AI crawlers forms a gradient: training crawlers < search crawlers < user-triggered fetchers
50%UnverifiedAt present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
50%UnverifiedBlocking GPTBot in order to appear in ChatGPT depends on which OpenAI agent you mean; if search visibility matters, you should consult the latest user agent documentation before writing rules.
50%UnverifiedOpenAI specifically defines crawlers such as GPTBot to decide which content can be read, for tools such as ChatGPT search.
50%UnverifiedTesting AI crawler reachability by spoofing user agents locally has limitations: some CDNs verify bots by IP, so a fake GPTBot may be blocked while the real one is allowed through, and local blocking results are therefore only a strong hint that must be confirmed with CDN logs and origin logs.
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedAttackers heavily forge the User-Agents of AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, with a single IP rotating through User-Agents such as KimiBot, Qwenbot and Hunyuan within a single session while requesting the `.env` file.
50%UnverifiedOpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.
50%UnverifiedThe user-agent is an HTTP request header field that clients use to self-report their identity, usually in a format such as 'Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)'.
50%UnverifiedAmong the 4,242 robots.txt files parsed in the September 28 snapshot, GPTBot was rejected 1.60 times for every one time it was allowed; for OAI-SearchBot this ratio was only 0.91.
50%UnverifiedMany websites' robots protocols block AI crawlers such as ClaudeBot and GPTBot, preventing these AI engines from accessing content on blocked sites.
50%UnverifiedOpenAI registered a dedicated crawler identity, GPTBot, and Google registered Google-Extended, allowing website administrators to selectively block them.
50%UnverifiedGPTBot, ClaudeBot, and Common Crawl are known AI training crawlers.
50%UnverifiedEmerging AI crawlers such as GPTBot, CCBot, and Amazonbot have flooded the web to collect training data in bulk, and some crawlers do not strictly comply with robots.txt restrictions or bypass filtering rules by disguising their User-Agent.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%