Google-Extended
A robots.txt user agent provided by Google that controls whether a website's content may be used for the training and improvement of Google AI models such as Gemini.
Timeline (last 90 days)
Google-Extended and Applebot-Extended are not crawlers but control tokens in robots.txt, used to decide whether content crawled by Googlebot and Applebot can be used for Gemini or Apple model training, and they do not appear in server logs
At present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
Googlebot and Google-Extended each serve different systems, and crawling behavior is separate from ranking and AI feature display.
AI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
Google-Extended controls whether Google can use content for Gemini model training and grounding; it does not affect Google Search or AI Overviews, which use the regular Googlebot.
Blocking Google-Extended does not affect a site's indexing, ranking, or display in Google Search, nor does it require resubmitting the sitemap.
Google-Extended is a robots.txt product token introduced by Google in September 2023 to control training and grounding of Google's Gemini models, and it is a completely separate token from Googlebot.
OpenAI registered a dedicated crawler identity, GPTBot, and Google registered Google-Extended, allowing website administrators to selectively block them.
Mainstream AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google DeepMind), and Bytespider (ByteDance).
All Facts (9)
Mainstream AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google DeepMind), and Bytespider (ByteDance).
60%UnverifiedGoogle-Extended and Applebot-Extended are not crawlers but control tokens in robots.txt, used to decide whether content crawled by Googlebot and Applebot can be used for Gemini or Apple model training, and they do not appear in server logs
50%UnverifiedAt present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.
50%UnverifiedGooglebot and Google-Extended each serve different systems, and crawling behavior is separate from ranking and AI feature display.
50%UnverifiedGoogle-Extended controls whether Google can use content for Gemini model training and grounding; it does not affect Google Search or AI Overviews, which use the regular Googlebot.
50%UnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).
50%UnverifiedGoogle-Extended is a robots.txt product token introduced by Google in September 2023 to control training and grounding of Google's Gemini models, and it is a completely separate token from Googlebot.
50%UnverifiedBlocking Google-Extended does not affect a site's indexing, ranking, or display in Google Search, nor does it require resubmitting the sitemap.
50%UnverifiedOpenAI registered a dedicated crawler identity, GPTBot, and Google registered Google-Extended, allowing website administrators to selectively block them.
50%