PerplexityBot
A web crawler operated by Perplexity that fetches web content to provide search and question-answering services.
Core Facts
Timeline (last 90 days)
GPTBot, PerplexityBot, and ClaudeBot are the crawlers used by OpenAI, Perplexity, and Anthropic, respectively, to crawl public web content.
PerplexityBot is Perplexity's indexing crawler, and fetches triggered by users' real-time questions may use the 'Perplexity-User' identifier.
blog.naver.com/robots.txt explicitly prohibits AI-related crawlers such as GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended, and states in a comment that crawling for AI training and RAG purposes is strictly forbidden.
A misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
Perplexity has two crawlers: PerplexityBot for displaying and linking websites in search results, and Perplexity-User for supporting user actions.
GPTBot (OpenAI training and retrieval), OAI-SearchBot (ChatGPT search), ClaudeBot, PerplexityBot, and Google-Extended (Google's AI training, independent of Googlebot) are currently the most critical AI crawler identifiers
GPTBot's server rejection rate was 14.4%, of which 11.1% occurred while robots.txt allowed it; ClaudeBot was rejected 14.8%, with 10.9% under allowance; PerplexityBot was rejected 11.3%, with 8.4% under allowance; OAI-SearchBot was rejected 9.8%, with 8.0% under allowance.
Among the 549 top-1000 Tranco sites that return 2xx to ordinary browsers, 78 (14.2%) rejected at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, even though their robots.txt explicitly allows these crawlers.
After excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.
Mainstream AI crawlers such as GPTBot, ClaudeBot and PerplexityBot currently largely do not execute JavaScript, so for sites that depend on JavaScript rendering, even if both robots.txt and the WAF let them through, the content they get is still empty.
6 more timeline events
All Facts (17)
AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and others.
65%UnverifiedGPTBot, PerplexityBot, and ClaudeBot are the crawlers used by OpenAI, Perplexity, and Anthropic, respectively, to crawl public web content.
50%UnverifiedPerplexityBot is Perplexity's indexing crawler, and fetches triggered by users' real-time questions may use the 'Perplexity-User' identifier.
50%Unverifiedblog.naver.com/robots.txt explicitly prohibits AI-related crawlers such as GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended, and states in a comment that crawling for AI training and RAG purposes is strictly forbidden.
50%UnverifiedA misspelling in the user-agent name will cause the rule to fail silently; the correct names include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User.
50%UnverifiedPerplexity has two crawlers: PerplexityBot for displaying and linking websites in search results, and Perplexity-User for supporting user actions.
50%UnverifiedGPTBot (OpenAI training and retrieval), OAI-SearchBot (ChatGPT search), ClaudeBot, PerplexityBot, and Google-Extended (Google's AI training, independent of Googlebot) are currently the most critical AI crawler identifiers
50%UnverifiedAmong the 549 top-1000 Tranco sites that return 2xx to ordinary browsers, 78 (14.2%) rejected at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, even though their robots.txt explicitly allows these crawlers.
50%UnverifiedAfter excluding the 14 sites rejected solely due to 429 rate limiting, 64 sites (11.7%) were still rejected by the server despite robots.txt allowing them.
50%UnverifiedMainstream AI crawlers such as GPTBot, ClaudeBot and PerplexityBot currently largely do not execute JavaScript, so for sites that depend on JavaScript rendering, even if both robots.txt and the WAF let them through, the content they get is still empty.
50%UnverifiedGPTBot's server rejection rate was 14.4%, of which 11.1% occurred while robots.txt allowed it; ClaudeBot was rejected 14.8%, with 10.9% under allowance; PerplexityBot was rejected 11.3%, with 8.4% under allowance; OAI-SearchBot was rejected 9.8%, with 8.0% under allowance.
50%UnverifiedIn a GEO workflow, pages must be crawlable by AI crawlers (such as GPTBot, ClaudeBot, PerplexityBot), otherwise they cannot be cited.
50%UnverifiedPerplexityBot's robots.txt compliance status is marked as disputed, with Perplexity claiming compliance but outside parties raising doubts; for Bytespider (ByteDance), no publicly released compliance policy was found
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedAttackers heavily forge the User-Agents of AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, with a single IP rotating through User-Agents such as KimiBot, Qwenbot and Hunyuan within a single session while requesting the `.env` file.
50%UnverifiedMarketers should check CDN or WAF logs to confirm that OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot receive 200 responses rather than 403.
50%UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools
50%