Unverified50% confidenceFactExact time
The default robots.txt generated by many hosting platforms or templates blanket-Disallows all crawlers, making content visible to Googlebot but completely invisible to AI engines
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/16/2026
First Seen
Valid until: 1/14/2027
Sources
Related Entities
Related Claims
VerifiedGooglebot determines the scope of crawlable pages by following each website's robots.txt protocol; robots.txt is a text file stored in the root directory of a website used to declare crawlable or blocked pages to web crawlers.78% similarUnverifiedGoogle has introduced the googlebot series of directive extensions, and some platforms support values such as noindex and nosnippet in the robots meta tag.73% similarUnverifiedGoogle can tolerate messy sites and execute JavaScript, while most AI crawlers do not have the same tolerance, which is one reason a site ranks well on Google but does not appear in AI answers.71% similarUnverifiedGoogle-Extended and Applebot-Extended are not crawlers but control tokens in robots.txt, used to decide whether content crawled by Googlebot and Applebot can be used for Gemini or Apple model training, and they do not appear in server logs71% similarUnverifiedGooglebot and Google-Extended each serve different systems, and crawling behavior is separate from ranking and AI feature display.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1044769API
curl https://kongchang.com/api/v1/knowledge/claims/1044769MCP
get_claim(id=1044769)