Unverified50% confidenceFactExact time
robots.txt User-agent matching is independent for each entry: a crawler first looks for the rule group whose name exactly matches its own UA, and only falls back to the User-agent: * wildcard rule when there is no match.
1
Sources
50%
Confidence
Long-term
Relevance
10/13/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedSome AI companies and search engines provide separate User-Agents for search indexing and AI training respectively, allowing webmasters to block training crawlers individually in robots.txt while retaining search indexing capability.69% similarUnverifiedMajor AI vendors state their crawlers' user-agent names and IP ranges in their official documentation, and the proper approach is to combine user-agent matching with source IP verification.67% similarUnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools61% similarUnverifiedAn Agent can verify whether code conforms to its own understanding, but cannot verify whether its understanding conforms to the user's expectations.60% similarUnverifiedMultiple bots in OpenBot perform task handoffs via @mentions.59% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1016002API
curl https://kongchang.com/api/v1/knowledge/claims/1016002MCP
get_claim(id=1016002)