User-Agent
A field in the HTTP request header used to identify information such as the browser engine, operating system and version that initiated the request. Servers use it to distinguish normal browsers from automated scripts.
Timeline (last 90 days)
A request sent to The Wall Street Journal returned 403, and merely changing the User-Agent to a Chrome browser string turned the same request into 401.
User-Agent is a field in the HTTP request header that the client uses to declare its own software identity, and it is entirely controlled by the client and can be forged arbitrarily.
Rotating only proxy IPs while keeping the same User-Agent and header set forms a stable fingerprint on the WAF side, and once that fingerprint is flagged, all IPs using it are implicated.
UA strings can be arbitrarily forged, so the mere appearance of GPTBot in logs cannot confirm the other party's identity; mainstream vendors have all published official IP ranges for verification.
User-Agent is a field in the HTTP request header, with a format like 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...', used to tell the server the browser engine, operating system, version and other information of the party making the request.
Using requests to request a website directly often fails to get the desired content, because the server determines whether the request comes from a real human browser or an automated program; the solution is to add headers and pass the browser's User-Agent into the request as a dictionary.
When the Vary header references a high-cardinality field such as User-Agent, cache effectiveness can be completely destroyed, with the cache hit rate approaching zero.
The User-Agent identifier in HTTP request headers explicitly declares the source of the request, enabling the server to distinguish AI agent traffic from regular browser traffic and implement targeted traffic management or blocking.
The server side can detect the User-Agent of AI crawlers and proactively return optimized Markdown content rather than full HTML
All Facts (9)
User-Agent is a field in the HTTP request header that the client uses to declare its own software identity, and it is entirely controlled by the client and can be forged arbitrarily.
50%UnverifiedRotating only proxy IPs while keeping the same User-Agent and header set forms a stable fingerprint on the WAF side, and once that fingerprint is flagged, all IPs using it are implicated.
50%UnverifiedUA strings can be arbitrarily forged, so the mere appearance of GPTBot in logs cannot confirm the other party's identity; mainstream vendors have all published official IP ranges for verification.
50%UnverifiedUser-Agent is a field in the HTTP request header, with a format like 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...', used to tell the server the browser engine, operating system, version and other information of the party making the request.
50%UnverifiedUsing requests to request a website directly often fails to get the desired content, because the server determines whether the request comes from a real human browser or an automated program; the solution is to add headers and pass the browser's User-Agent into the request as a dictionary.
50%UnverifiedWhen the Vary header references a high-cardinality field such as User-Agent, cache effectiveness can be completely destroyed, with the cache hit rate approaching zero.
50%UnverifiedThe User-Agent identifier in HTTP request headers explicitly declares the source of the request, enabling the server to distinguish AI agent traffic from regular browser traffic and implement targeted traffic management or blocking.
50%UnverifiedThe server side can detect the User-Agent of AI crawlers and proactively return optimized Markdown content rather than full HTML
50%UnverifiedA request sent to The Wall Street Journal returned 403, and merely changing the User-Agent to a Chrome browser string turned the same request into 401.
50%