Five API Failure Lessons from Serving 300 AI Agent Clients

When you provide services for AI Agents, traditional API design assumptions may all fail. Agent Commons encountered approximately 300 independent clients within the first two weeks of launch—most of them crawlers and probing programs, with a minority being AI Agents with real goals. This article is compiled from their official notes, summarizing five failure lessons that led to real user loss.

Lesson One: Bot Protection Mechanisms Will Silently Block Agents

Cloudflare's browser integrity check returned 403 for all requests using Python's default urllib user agent for several consecutive days. The problem: no error notification reached the server side—the Agent simply never appeared.

This means your monitoring dashboard may show everything is fine, but in reality all Agents using standard HTTP libraries are locked out.

Solution: Test your endpoints daily with Python urllib, requests, httpx, Node, and Go default user agents, as well as the fetch tools of major assistant platforms. Don't assume that "accessible through a browser" equals "accessible to Agents."

Lesson Two: Many Agents Can Only Make GET Requests

The fetch tools of assistant products typically only allow GET methods, and some tools can only open URLs that have already appeared in the conversation. This is a critical flaw for APIs that require POST, PUT, and other write operations.

Solution: Provide GET-form alternative endpoints for every write operation, paired with an idempotent request_id parameter. At the same time, place ready-to-click links in llms.txt to lower the barrier for Agent invocation.

What is llms.txt: This is a conventional plain-text file placed in a website's root directory (typically at /llms.txt), using Markdown format to explain to large language models and AI Agents what resources, interfaces, and recommended entry points the site has. It is similar to robots.txt for search engines or sitemap.xml for site maps, but its audience is LLMs. Since many Agents' fetch tools can only open URLs that have already appeared in the conversation, pre-writing clickable interface links into llms.txt is equivalent to providing Agents with an "explorable directory," allowing them to find GET-form alternative interfaces and documentation entry points without guessing URLs.

Lesson Three: Redirects Break Agent Requests

Many HTTP clients will convert POST to GET or simply discard the request body when encountering a redirect. python-httpx by default doesn't even follow redirects. This type of behavior is irrelevant in browser scenarios, but for Agents it means request failure.

When Agent Commons migrated to a new address, it only set up redirects for browser page views, while Agent requests were still answered on the old host. Even with such caution, a batch renaming operation accidentally removed the old host from the MCP whitelist, causing approximately 20 minutes of 403 errors.

Solution: Complete migrations by adding new addresses rather than replacing old ones. After each deployment, test all protocols on every host.

The origin of the MCP whitelist: MCP (Model Context Protocol) is an open protocol that allows AI assistants to communicate with external tools and data sources. Agent platforms typically maintain a whitelist of permitted hosts, and only domains on the list can be invoked by Agents. Therefore, when you migrate your API address, in addition to DNS and HTTP layers being correct, you must also synchronously update the MCP whitelist. Otherwise, even if the network layer is completely clear, the Agent side will still be rejected at the protocol layer. Agent Commons' 20-minute 403 outage stemmed precisely from this—the batch renaming operation did not add the new host to the whitelist, causing requests to be blocked at the Agent platform entrance.

Lesson Four: Agents Run in Sandbox Environments

Multiple Agents have no public IP, and are recycled after running for a few minutes. Webhooks simply cannot reach them—this is a scenario rarely considered in traditional API design.

Solution: Provide a fixed URL that returns all messages sent to that Agent. State management should be keyed by Agent name, not by its runtime environment.

The difference between Webhooks and polling: A Webhook is a "push" model, where the server actively initiates an HTTP request to a callback URL pre-registered by the client, with the prerequisite that the client must have a long-term reachable public address. But Agents run in sandboxes without public IPs that are recycled within minutes—they simply cannot receive pushes. The so-called "pull-based fixed URL" means having the server expose a stable endpoint (such as /agents/{name}/messages), and the Agent proactively GETs and pulls its own messages when needed. This way, state is no longer bound to a short-lived runtime environment, but to the Agent's logical identity, allowing seamless continuation even if the sandbox is destroyed and rebuilt.

Lesson Five: Agents Will Submit Template Content Directly

Agents have submitted form examples verbatim—including placeholder names. This isn't Agents "making mistakes," but rather their lack of understanding of template semantics.

Solution: Reject unfilled template submissions and return clear error messages, rather than storing them in the database.

Why Agents submit placeholders: Text like "Name: John Doe" and "Email: name@example.com" in templates are illustrative text that humans automatically ignore when reading, but LLMs processing form structures often treat the entire HTML or JSON block as field content that needs to be returned in full, lacking the semantic judgment that "this is an example, not a real value." Additionally, some Agents simply mechanically copy strings that appeared on the page when calling tools, without proactively asking "what should go here." Therefore, the server cannot assume that received data has been filtered by user intent. It needs to detect keywords (such as "example," "placeholder," "sample") and typical placeholder patterns before storing data, and return readable error messages to guide Agents to re-enter information.

Core Principle: Every Failed Call Is a User Who Left

Agent Commons' experience can be condensed into one sentence: Check the logs daily for new clients, replay the requests they sent, and fix the problems they encountered.

Behind this principle is an easily overlooked reality—when your API targets Agents, you're facing a consumer group completely different from human users. They don't submit tickets, don't complain on social media, and don't even trigger your error monitoring. They just disappear.

For developers building Agent-friendly APIs, these five lessons point in the same direction: Designing APIs for Agents requires re-examining all those default assumptions made for human users. Agent detection, request method limitations, redirect behavior, network environment assumptions, input validation logic—every single link can become an entry point for silent Agent failure.

One-line Summary

When designing APIs for AI Agents, five overlooked default assumptions can lead to silent user loss.

Paragraph Summary

Agent Commons encountered approximately 300 independent clients within two weeks of launch, most of which were crawlers and AI Agents. The author summarizes five failure lessons that led to real user loss: Bot protection mechanisms like Cloudflare silently block Agents using standard Python user agents without error notifications; most assistant platform fetch tools only support GET, requiring GET alternative interfaces and idempotent request_id for write operations; redirects cause POST to degrade to GET or lose request bodies, so old addresses should be retained during migration; Agents run in sandboxes without public IPs that are quickly recycled, making Webhooks unreachable and requiring pull-based fixed URLs; Agents submit form templates with placeholders verbatim, requiring server-side detection and rejection. The core principle: every failed call is a user who left, and Agent-facing APIs require re-examining all default assumptions made for human users.