Unverified50% confidenceFactExact time
The industry is discussing extending robots.txt semantics, for example clearly distinguishing different purposes such as "search crawling", "training data crawling" and "agent acting on behalf of a user", along with the IETF AIPREF working group and others developing machine-readable preference expressions.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/17/2026
First Seen
Valid until: 1/15/2027
Sources
Related Entities
Related Claims
UnverifiedThe access permissions set in robots.txt for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot directly determine whether content can enter these models' training sets or real-time retrieval pools68% similarUnverifiedSome AI companies and search engines provide separate User-Agents for search indexing and AI training respectively, allowing webmasters to block training crawlers individually in robots.txt while retaining search indexing capability.68% similarUnverifiedRobot sports competitions provide an open and standardized platform for technical comparison among different manufacturers and research teams.67% similarUnverifiedIn the long term, when Agents move from experimentation to production, compute demand will shift from "accepting requests" to "running continuously", which is a completely different load pattern from chatbots.66% similarUnverifiedAI crawler user agents can be divided into three categories: training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User).65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1053191API
curl https://kongchang.com/api/v1/knowledge/claims/1053191MCP
get_claim(id=1053191)