Vercel Firewall New Feature: Manage AI Crawlers and AI Assistants Separately

Vercel Firewall recently rolled out a noteworthy update: within the AI Bots managed ruleset, users can now separately log or block requests from AI crawlers and AI assistants. This resolves a dilemma that site owners previously faced—wanting to prevent content from being scraped for free for model training, while not wanting to affect users accessing real-time information through AI assistants.

Why Distinguish Between the Two Types of AI Traffic

Vercel explicitly divides AI traffic into two categories:

• AI crawlers: Bots that collect content for indexing and model training purposes.
• AI assistants: Bots that scrape pages in response to user actions or search queries. The typical scenario is a user asking an AI a question, and the assistant fetching web content in real time to compose an answer.

The business implications of these two types of traffic are entirely different. Crawler scraping typically doesn't directly bring return visits, and after content is used for training it may replace the original site; whereas AI assistant scraping is often triggered by real user questions and represents a potential traffic entry point. In the past, many protection solutions could only indiscriminately block all AI bots, causing sites to lose the opportunity to be cited by AI assistants while blocking training scraping. Vercel's move to separate the two allows site owners to formulate strategies as needed.

Bot Management is a general capability within firewall products: the platform maintains an identification list of known bots, classifies requests based on signals such as User-Agent, IP ranges, and behavioral characteristics, and then applies actions like allow, log, or block to each category. A managed ruleset means that this list and the decision logic are maintained and updated uniformly by the platform—sites only need to choose a strategy without writing their own matching conditions. The value of this model lies in coping with the rapid changes in the bot ecosystem—new AI scrapers keep emerging, and self-built lists quickly become outdated. The trade-off is limited transparency in decision-making; sites have difficulty knowing precisely why a request was classified as a crawler or assistant, so in cases of misclassification they typically can only rely on platform corrections or fall back to custom rules.

How to Configure the AI Bots Ruleset

The configuration entry is on the project's Security > Firewall > Rules page. Under the Bot Management section, enable AI Bots, click the gear icon next to it, and you can set actions separately for AI crawlers and AI assistants. After setting up, review the changes before publishing.

A typical strategy combination is: block AI crawlers while allowing and logging AI assistant requests. This protects content from being used on a large scale for training while retaining the potential exposure brought by AI assistants.

Command Line Also Supported

For developers accustomed to using the CLI, Vercel also provides corresponding commands. For example, to block AI crawlers:

vercel firewall rules edit ai-bots --category ai-crawlers --action deny --yes

To allow and log AI assistant requests:

vercel firewall rules edit ai-bots --category ai-assistants --action log --yes

After modification, you can use vercel firewall diff to view changes, and once confirmed, publish them via vercel firewall publish. This workflow incorporates configuration into a versioned process, suitable for team collaboration and CI/CD scenarios.

The difference between --action log and --action deny is worth explaining: deny directly rejects the request and returns an error response, so the bot cannot obtain content; log lets the request through but records the event, making it easy to observe the scale, sources, and access paths of a certain type of traffic in the firewall logs without affecting actual access. In practice, it's common to first use log to observe for a period, assess misclassification risk and traffic proportion, and then decide whether to tighten to deny. Similarly, vercel firewall diff shows the difference between the local configuration and the live effective configuration, and vercel firewall publish pushes the changes to the production environment. This two-stage "edit—review—publish" process allows rule changes to be reviewed and rolled back like code.

Availability and Maintenance

The AI Bots managed ruleset is available to all plans at no extra charge. Vercel will continuously update the rules as new bots emerge, which means site owners don't need to manually maintain an ever-growing bot list. For content sites, e-commerce, and SaaS products, this capability provides a more granular means of traffic governance—finding a balance between openness and protection.

From an industry trend perspective, the boundary between AI scraping and content licensing is still being contested. Vercel delegates control to the site level rather than making uniform decisions for users—this approach is worth noting. What's worth watching going forward: whether the ruleset can cover enough AI bots, whether the classification judgments are accurate, and how sites weigh the trade-offs between the two types of traffic in actual use.

One-line Summary

Vercel Firewall supports separately logging or blocking AI crawler and AI assistant traffic.

Paragraph Summary

Vercel Firewall's AI Bots managed ruleset adds category-based control, allowing site owners to separately set log or block actions for AI crawlers and AI assistants. AI crawlers collect content for indexing and model training and typically don't bring return visits; AI assistants trigger real-time scraping from real user questions and are a potential traffic entry point. Previously, protection solutions often indiscriminately blocked all AI bots, causing sites to lose the chance of being cited by AI assistants while blocking training scraping. Users can configure in the console under Security > Firewall > Rules, or manage rules via Vercel CLI in a versioned workflow—first diff for review, then publish to take effect. The ruleset is available to all plans and free, and Vercel will continuously update the bot list.