OAI-SearchBot
A crawler operated by OpenAI that fetches web content for the ChatGPT search feature. It is distinguished from GPTBot, the crawler used for collecting training data.
Core Facts
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
OpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.
Timeline (last 90 days)
Among the 549 top-1000 Tranco sites that return 2xx to ordinary browsers, 78 (14.2%) rejected at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, even though their robots.txt explicitly allows these crawlers.
You can compare the status codes returned by curl when requesting the homepage with a Mozilla browser user agent versus an OAI-SearchBot user agent to determine whether server behavior is consistent with robots.txt: a browser getting 200 and a crawler getting 403 indicates that a CDN or WAF layer rule is at work.
Blocking GPTBot (training) while allowing OAI-SearchBot (ChatGPT search citations) is a reasonable and common choice, and the 2.6% of top sites that seriously do this prove it works.
GPTBot's server rejection rate was 14.4%, of which 11.1% occurred while robots.txt allowed it; ClaudeBot was rejected 14.8%, with 10.9% under allowance; PerplexityBot was rejected 11.3%, with 8.4% under allowance; OAI-SearchBot was rejected 9.8%, with 8.0% under allowance.
After excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.
OpenAI states that websites opting out of OAI-SearchBot will not appear in ChatGPT search answers, but may still appear in the form of navigational links
The opt-out cost for the three types of AI crawlers forms a gradient: training crawlers < search crawlers < user-triggered fetchers
Blocking GPTBot in order to appear in ChatGPT depends on which OpenAI agent you mean; if search visibility matters, you should consult the latest user agent documentation before writing rules.
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
OpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.
3 more timeline events
All Facts (14)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%Partially VerifiedOpenAI's official documentation states that settings such as OAI-SearchBot, GPTBot, and ChatGPT-User are independent of one another; OAI-SearchBot is responsible for making sites appear in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.
70%UnverifiedYou can compare the status codes returned by curl when requesting the homepage with a Mozilla browser user agent versus an OAI-SearchBot user agent to determine whether server behavior is consistent with robots.txt: a browser getting 200 and a crawler getting 403 indicates that a CDN or WAF layer rule is at work.
50%UnverifiedAmong the 549 top-1000 Tranco sites that return 2xx to ordinary browsers, 78 (14.2%) rejected at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, even though their robots.txt explicitly allows these crawlers.
50%UnverifiedAfter excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.
50%UnverifiedGPTBot's server rejection rate was 14.4%, of which 11.1% occurred while robots.txt allowed it; ClaudeBot was rejected 14.8%, with 10.9% under allowance; PerplexityBot was rejected 11.3%, with 8.4% under allowance; OAI-SearchBot was rejected 9.8%, with 8.0% under allowance.
50%UnverifiedBlocking GPTBot (training) while allowing OAI-SearchBot (ChatGPT search citations) is a reasonable and common choice, and the 2.6% of top sites that seriously do this prove it works.
50%UnverifiedThe opt-out cost for the three types of AI crawlers forms a gradient: training crawlers < search crawlers < user-triggered fetchers
50%UnverifiedOpenAI states that websites opting out of OAI-SearchBot will not appear in ChatGPT search answers, but may still appear in the form of navigational links
50%UnverifiedBlocking GPTBot in order to appear in ChatGPT depends on which OpenAI agent you mean; if search visibility matters, you should consult the latest user agent documentation before writing rules.
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedMarketers should check CDN or WAF logs to confirm that OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot receive 200 responses rather than 403.
50%UnverifiedAmong the 4,242 robots.txt files parsed in the September 28 snapshot, GPTBot was rejected 1.60 times for every one time it was allowed; for OAI-SearchBot this ratio was only 0.91.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%