Unverified50% confidenceConceptExact time
The same robots.txt can finely control allowing search indexing but prohibiting AI training, because different AI vendors use multiple different User-agents corresponding to different purposes.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedMany mainstream AI training datasets were formed without strictly adhering to the robots.txt protocol.78% similarVerifiedMany AI training data crawlers tend to ignore the robots.txt protocol and rapidly scrape entire sites at extremely high concurrency, fundamentally differing from traditional search engine crawlers.78% similarUnverifiedWhen an AI Agent calls external tools, subtle changes in search query parameters (such as changing from 'machine learning best practices' to 'machine learning practice guide') can produce completely different retrieval results.74% similarUnverifiedAI agent execution frameworks (such as LangChain, AutoGPT, or OpenAI's Assistants API) do not by default enforce checking of robots.txt, unless developers actively implement the relevant logic.74% similarUnverifiedAI crawlers can be divided by purpose into three categories: training crawlers, search crawlers, and user-triggered fetchers, each with its own independent robots.txt identifier74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1048788API
curl https://kongchang.com/api/v1/knowledge/claims/1048788MCP
get_claim(id=1048788)