Applebot-Extended
A robots.txt user agent provided by Apple that controls whether a website's content may be used for the training of Apple's AI models, such as Apple Intelligence.
Timeline (last 90 days)
Google-Extended and Applebot-Extended are not crawlers but control tokens in robots.txt, used to decide whether content crawled by Googlebot and Applebot can be used for Gemini or Apple model training, and they do not appear in server logs
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
A similar product-token concept also appears at other vendors: Applebot-Extended controls Apple Intelligence training, and Bytespider (ByteDance) serves both search and training purposes.
All Facts (3)
Google-Extended and Applebot-Extended are not crawlers but control tokens in robots.txt, used to decide whether content crawled by Googlebot and Applebot can be used for Gemini or Apple model training, and they do not appear in server logs
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedA similar product-token concept also appears at other vendors: Applebot-Extended controls Apple Intelligence training, and Bytespider (ByteDance) serves both search and training purposes.
50%