AI Agents Visiting Websites: Yelp Charges, JPMorgan Blocks—Who Should Decide Access?

When AI agents start browsing websites, scraping information, and even executing transactions like human users, a thorny question surfaces: should websites let them in? According to an RSS report, Yelp has chosen to charge AI agents for access, JPMorgan has explicitly refused, and Shopify has turned the decision into a switch. Behind these three starkly different attitudes lies a long-overlooked fact—admission is a business decision, not something bot rules should govern.

Three Attitudes: Charge, Refuse, Hand It to Users

Yelp's approach is the most direct: treat AI agent access as a business and charge for it. This amounts to acknowledging that agent traffic has commercial value—better to monetize it than block it. JPMorgan takes the opposite stance, refusing AI agents entry into its systems, driven by strong concerns over financial data security and compliance risks. Shopify's path is more subtle—it doesn't decide for merchants, but turns "whether to allow AI agent access" into a configurable switch, letting each store choose for itself.

These three attitudes are not simply a dispute over technical approaches, but different business answers to the same question: is the traffic brought by AI agents an asset or a liability?

Bot rules typically refer to the robots.txt protocol and similar crawler management conventions. This unofficial standard, drafted in 1994, uses a plain-text file in a website's root directory to tell search engine crawlers which paths can be crawled and which should be avoided. It relies on crawlers' voluntary compliance and has no legal force. Its design assumption is that visitors are homogeneous, identifiable crawlers, and that crawling behavior itself doesn't distinguish commercial intent. For this reason, robots.txt can only express a binary "allow/deny" attitude—it cannot distinguish whether an agent represents real user behavior, the load cost of access frequency, or the potential transactional value behind access. As AI agents increasingly resemble human users and behavioral boundaries blur, this static rule set proves inadequate for carrying commercial admission judgments.

Core Argument: Admission Is a Business Decision, Not Bot Rules

The original article's judgment is sharp: admission is a business decision, and bot rules are the wrong enforcement venue. This punctures a common industry misconception—many websites try to use robots.txt or similar anti-scraping protocols to manage AI agents, but these technical rules are inherently incapable of carrying commercial intent.

Bot rules were designed to tell crawlers "which pages can be scraped." They cannot answer questions like "does this agent represent a real user," "is the value it brings worth admitting," or "what risks might admission entail." Stuffing these judgments into technical rules results in either blanket bans that harm valuable agent traffic, or rules that exist in name only and become mere formalities.

Why Technical Rules Can't Manage Commercial Admission

Anti-scraping protocols are static and global, while commercial admission is dynamic and context-specific. An e-commerce company might be willing to let price-comparison agents in to increase exposure, but unwilling to let inventory-scraping agents in; a bank might allow customer-service agents to query public information, but refuse any agent involving account operations. This granularity of difference simply cannot be expressed through uniform bot rules.

The key difference between AI agents and traditional crawlers lies in agency and intent. Traditional crawlers typically scrape in batches at fixed frequencies and fixed paths, with easily identifiable behavior patterns; AI agents, often driven by large language models, can execute multi-step tasks, simulate clicks and form filling, and even complete orders, price comparisons, and customer service conversations. This means the same type of agent may simultaneously generate "beneficial traffic" (e.g., bringing genuine purchase intent) and "harmful load" (e.g., high-frequency requests overwhelming services). Moreover, agent identity attribution (which user, which platform it represents) is often invisible at the HTTP layer, making it hard for websites to judge commercial attributes from request characteristics alone. Therefore, admission management for AI agents requires higher-level mechanisms such as identity authentication, access authorization, and behavior auditing—not just User-Agent identification or IP blocking.

Implications for Website Operators

For website operators currently facing AI agent traffic, these three cases offer a referable decision framework.

Classify First, Then Price

Not all AI agents are the same. Information-query, price-comparison, transaction, and automated-operation agents each have different value and risk to websites. Yelp's charging model works because it knows which agent visits are worth charging for. Operators need to establish an agent classification system before discussing admission strategies.

Hand the Switch to the Business Side

Shopify's approach is worth emulating: the technical team provides the capability, the business side makes the decision. Rather than having engineering teams uniformly block or allow at the server level, it's better to make agent admission a productized configuration item, letting the teams most familiar with their own business risks set the rules.

Account for Risk and Reward Separately

JPMorgan's refusal is not conservatism but a clear-eyed calculation of risk versus reward. In financial scenarios, a single data breach or non-compliant operation caused by an agent carries costs far exceeding the convenience brought by agent traffic. Websites need to evaluate the potential benefits and potential losses of agent access separately based on their industry attributes, rather than applying a uniform "open" or "closed" stance.

Agent identity authentication and access authorization are emerging as new infrastructure directions. OAuth-like authorization frameworks, Verifiable Credentials, and access protocols specifically for AI agents (such as the agent auth standards proposed by some vendors) attempt to solve the problem of "who this agent represents and what permissions it's granted." On this basis, value settlement mechanisms (e.g., per-call billing, revenue sharing based on conversions) turn agent traffic from an unmeasurable cost item into a priceable commodity. This is consistent with Yelp's charging model but more systematic. For website operators, following the evolution of these standards has more long-term value than continuously patching robots.txt.

Industry Direction: From Technical Confrontation to Business Negotiation

The current situation is more like a transitional period. On one side are Yelp-style commercialization attempts, on the other JPMorgan-style firm refusals, with Shopify-style configurable solutions in between. As the number of AI agents grows, the game between websites and agents will shift from cat-and-mouse at the technical layer to rule negotiation at the business layer—agent identity authentication, access authorization, and value settlement may all become new infrastructure.

The real question is no longer "should we let AI agents in" but "under what conditions, within what scope, and who decides to let them in." Returning this question to business logic, rather than hiding it in the corner of bot rules, is the direction the industry needs to take.

One-line Summary

AI agent access to websites should be treated as a business decision, not a technical issue that bot rules can govern.

Paragraph Summary

Using three cases—Yelp charging for access, JPMorgan refusing, and Shopify turning it into a configurable switch—the article reveals three attitudes websites take toward AI agents. The core argument: admission is inherently a business decision, and bot rules (like robots.txt) are static and global, unable to carry dynamic, context-specific commercial intent. The article advises operators to classify agents before pricing, hand admission switches to the business side, and evaluate risk and reward separately. The industry is shifting from technical confrontation to business negotiation, with agent identity authentication and value settlement potentially becoming new infrastructure.