Vercel Observability Adds AI Traffic Insights: See How AI Crawlers Access Your Site

Vercel has added AI traffic insights to its Observability product, giving developers the ability to directly query how AI bots access their sites. This update brings a long-opaque traffic source—AI crawlers and assistant requests—into the standard observability framework, which is particularly relevant for sites that rely on content distribution.

From "Invisible" to "Queryable by Group"

In the past, site access logs were a mix of search engine crawlers, regular users, and a growing number of AI bots. You roughly knew AI traffic existed, but it was hard to answer "which bot came" or "which pages did it crawl." Vercel's new feature separates these requests out and displays them on the AI Traffic page in Observability.

The entry point is in the Vercel console under Observability > AI Traffic. In addition to viewing in the UI, you can query these metrics directly via the Vercel CLI vercel metrics command, for example, grouping request counts by bot name:

vc metrics vercel.ai_traffic.count --group-by botName

Note that CLI queries require an Observability Plus subscription. This means there's a permission distinction between basic UI viewing and more granular programmatic queries, and teams need to decide whether to upgrade based on their observability needs.

What Each of the Five Core Metrics Solves

The official AI traffic insights include five dimensions, each addressing a specific operational question:

Verified AI bot request count

Counts requests by each verified AI bot. This solves the "who is crawling" question, helping you distinguish the crawling intensity of different bots and determine which AI systems are more interested in your content.

"Verified AI bots" typically refer to crawlers whose identity is confirmed via reverse DNS or IP range verification, such as OpenAI's GPTBot, Google's Google-Extended, Anthropic's ClaudeBot, etc. The purpose of this verification mechanism is to distinguish real AI vendor crawlers from ordinary scrapers masquerading as AI bots, avoiding polluted statistics. Vercel categorizing these requests separately means its underlying system already performs identity recognition on request sources; the botName developers see is an attributable identifier, not simple User-Agent string matching.

Most-crawled pages

Shows the pages most frequently requested by AI crawlers, covering both indexing and model training purposes. For content sites, this is direct evidence of which pages are actually "seen" by the AI ecosystem.

AI crawlers typically crawl pages for two purposes: one is to build indexes for search engines or AI search, used for subsequent retrieval and citation; the other is to collect corpus for model training. These two purposes have different impacts on sites—indexing crawls focus more on content timeliness and discoverability, while training crawls focus more on long-term value and licensing strategy. Vercel's most-crawled pages metric combines both purposes; if developers need further distinction, they can use botName group queries to determine which AI system is crawling which pages.

Assistant retrievals

Records pages crawled by AI assistants when responding to user actions or search queries. This metric differs from crawler crawling—it reflects real-time, Q&A-style access behavior, closer to "how AI is citing your content to answer users."

The key difference between assistant retrieval and regular crawler crawling lies in trigger timing and purpose. Crawler crawling is typically periodic and batch-oriented, aimed at building or updating indexes; assistant retrieval is on-demand crawling triggered in real-time when a user initiates a query, aimed at finding answer sources for the current question. This access pattern is more sensitive to site response speed and content structure, and better reflects actual content citation in Q&A scenarios. Traditional web analytics tools usually mix such requests into general traffic, making them hard to observe separately.

Markdown requests

Counts pages requested by AI bots in Markdown format. As more AI tools prefer structured, easily parseable content formats, this dimension reflects whether your site is being read in a way more suitable for model consumption.

Markdown format requests reflect an emerging content consumption pattern: some AI tools and clients, when crawling web pages, preferentially request the Markdown version rather than rendered HTML, because Markdown structure is clearer, more token-efficient, and easier for models to parse directly. Whether to provide a Markdown version involves site architecture decisions, such as using content negotiation to return different formats at the same URL based on the Accept header, or maintaining a separate .md endpoint. This metric helps teams assess the actual scale of such requests to determine whether the investment is worthwhile.

Errors

Lists pages where AI bot requests returned errors. Errors during AI crawling often mean that content is effectively missing from the AI ecosystem—an easily overlooked but directly impactful issue.

Practical Value for Developers and Content Teams

This feature isn't just "another monitoring dashboard"—it brings AI traffic into the realm of attributable, troubleshootable data.

For developers, grouping queries by botName means AI traffic can be separated from general traffic for capacity planning and error troubleshooting. If requests from a certain type of bot grow abnormally, or errors cluster on certain pages, you can more quickly determine whether it's a crawling strategy issue or a site configuration issue.

For content and SEO teams, most-crawled pages and assistant retrievals offer new perspectives. Traditional analytics tools excel at tracking human visits and search rankings, but lack direct data on "which content is cited by AI assistants." These two metrics fill that gap, allowing content strategy to target both human readers and AI consumption scenarios.

The Markdown requests metric points to a more specific technical decision: whether the site should provide Markdown versions of content for AI clients. This metric provides evidence based on actual request volume rather than guesswork.

A Few Reminders Before Use

The entry points are clear, but to get value you should note: CLI queries depend on Observability Plus, so teams need to evaluate subscription costs; the five metrics each answer different questions, so it's advisable to first clarify whether you're solving crawl attribution, content coverage, or error troubleshooting, then look at the corresponding dimensions accordingly. Also, the official documentation provides more complete explanations and is worth reading through before integration.

AI traffic is becoming an important part of site traffic, and separating it from log noise is the first step toward refined operations. Vercel's update provides exactly such a starting point.

One-line Summary

Vercel Observability adds an AI Traffic feature, letting developers query how AI bots access their sites.

Paragraph Summary

Vercel has added an AI Traffic page to its Observability product, separating AI crawler and assistant requests from mixed access logs. It provides five core metrics: verified AI bot request counts, most-crawled pages, assistant retrievals, Markdown requests, and error pages. Developers can view them in the console UI or query by botName via the Vercel CLI vercel metrics command, though CLI queries require an Observability Plus subscription. The value for developers is bringing AI traffic into an attributable and troubleshootable scope; for content and SEO teams, it fills the gap in traditional analytics regarding "which content is cited by AI," and the Markdown requests metric provides data to decide whether to offer Markdown versions of content.