Unverified50% confidenceFactExact time
Under default configurations, many CDNs/WAFs place known AI crawlers into a recognized bot category and apply a challenge (returning 403, 429 or a JS challenge page) rather than letting them through directly.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/13/2026
First Seen
Valid until: 1/11/2027
Sources
Related Entities
Related Claims
UnverifiedWAFs typically do not return a direct 403 for crawler traffic, but instead deliver challenge pages such as JS Challenge, CAPTCHA, and Turnstile, causing clients to continuously hit 503 or 429.66% similarUnverified多数AI爬虫和Agent不会主动发送Accept: text/markdown请求头，而是沿用通用请求头或抓取原始HTML59% similarUnverifiedAI爬虫（如GPTBot、ClaudeBot、Common Crawl）采用轻量级HTTP请求+HTML解析方式，不为每个页面启动完整浏览器执行JavaScript58% similarUnverifiedAfter adding a Webhook to a Bot, the Bot details page provides a request address and a key used for identity verification, and the caller must put the key into the request header or signature parameter as officially required.57% similarUnverifiedAI search crawlers generally do not execute JavaScript or execute only very limited scripts, so what they see is the initial HTML returned by the server; the body text of client-side rendered (CSR) pages may be empty in the crawler's eyes.54% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1020799API
curl https://kongchang.com/api/v1/knowledge/claims/1020799MCP
get_claim(id=1020799)