Unverified50% confidenceFactExact time
多数AI爬虫和Agent不会主动发送Accept: text/markdown请求头，而是沿用通用请求头或抓取原始HTML
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
8/28/2026
First Seen
Valid until: 11/26/2026
Sources
Related Entities
Related Claims
UnverifiedAI爬虫（如GPTBot、ClaudeBot、Common Crawl）采用轻量级HTTP请求+HTML解析方式，不为每个页面启动完整浏览器执行JavaScript71% similarUnverifiedTraditional HTTP request-based crawlers (such as Python's requests library) can only retrieve raw HTML returned by the server and cannot execute JavaScript, thus failing to see dynamically loaded content.60% similarUnverifiedA crawler program sends HTTP requests to a server via code, and the server returns raw content; however, since the crawler lacks a browser's rendering capability, what it receives is the raw HTML source code of the webpage60% similarUnverified浏览器中 HTMLElement 继承自 Element，Element 继承自 Node，反爬脚本可通过 Object.getOwnPropertyDescriptor() 或 instanceof 检查识别伪造对象58% similarUnverifiedFor non-simple requests (such as those containing custom headers or using PUT/DELETE methods), the browser sends a preflight request (using the OPTIONS method) to confirm whether the server allows the actual request.57% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/812386API
curl https://kongchang.com/api/v1/knowledge/claims/812386MCP
get_claim(id=812386)