Meta Accuses Claude Tool of Crawling at Excessive Speed: 124 Lines of Python Set Rules for AI Agents

Title

Meta Accuses Claude Tool of Crawling at Excessive Speed: 124 Lines of Python Set Rules for AI Agents

Meta Description

Meta accused a Claude-related tool of crawling Instagram at four times human speed, causing an erroneous ban. This article analyzes a 124-line standard-library Python guardrail solution covering token buckets, jitter, Retry-After backoff, robots.txt, honest User-Agent, daily caps, and JSONL audit logs, with real test output and a pre-deployment checklist.

Article Content

A minor incident between Meta and Anthropic has inadvertently exposed a long-overlooked problem in AI agent deployment: when a model calls browser tools in a loop, there is no mechanism to make it behave like a "polite" visitor. This article introduces a 124-line standard-library Python solution to rein in the HTTP behavior of AI agents.

Incident Recap: A Mechanism Discussion Triggered by an Erroneous Ban

According to reports from The Next Web and AI Weekly, Instagram briefly banned the No Kings Coalition account for about 90 minutes. Meta's VP of Communications Andy Stone later explained that a tool related to Anthropic's Claude accessed the coalition's profiles, posts, and hashtags at roughly four times human speed, and the short, dense request pattern closely resembled automated bot traffic. Meta acknowledged the ban was an error and apologized, but did not disclose which specific tool was involved, who ran it, or for what purpose.

The incident itself still has unresolved aspects, but the core fact corroborated by multiple sources is: platform risk-control systems don't care whether traffic comes from a "helpful assistant"—they only look at the shape of burst requests and take action against accounts or IPs. This is exactly what all agent projects involving web access need to confront.

Why "Politeness" Needs to Be Written into Code

The author's view is direct: the model itself has no concept of "slowing down," and browser or HTTP tools don't proactively rate-limit either—what's missing is a middleware layer between them that understands throttling. He calls this middleware the "guardrail layer," composed of seven small mechanisms:

• Per-host token bucket: Each site has a steady request pace (default 12 per minute, burst of 2), preventing the model from firing dozens of requests at once when it suddenly decides to "check all profiles."
• Random jitter: Humans don't click precisely every 5.000 seconds; a fixed interval is itself a bot signature.
• 429 and 503 backoff: When the server returns Retry-After, wait strictly as required (supporting both seconds and HTTP date formats); otherwise use jittered exponential backoff.
• robots.txt: Check with urllib.robotparser before requesting; Crawl-delay lowers the token bucket rate.
• Honest User-Agent: Identify as a bot and provide a contact URL and email so site operators can reach you instead of blocking the entire IP range.
• Daily action cap: A hard ceiling that runaway loops cannot breach; retries count too, because the server sees them as well.
• Append-only action log: One line of JSON per action, including rejected requests, so you can answer "what did the agent actually do" in seconds when problems arise.

A Few Design Decisions Worth Explaining

The token bucket allows "overdraft": when two threads take tokens simultaneously, the second sleeps longer, keeping lock hold times short while maintaining correct pacing. If the server's required wait exceeds max_backoff, the client chooses to give up rather than retry early—retrying before Retry-After expires is precisely the behavior that escalates rate limiting into a ban. The daily cap is rebuilt from the log at startup; the author's first version kept the count in memory, which reset to zero on a crash-restart, and only reading the log made the cap real.

The handling of robots.txt fetch failures is also conservative: a 404 means "no rules," but a network failure means "I don't know," and in that case everything is disallowed.

The author also shared a bug found during testing: writing self.today, self.count = utc_day(), self._count_today() in __init__—Python evaluates the entire right side first, causing _count_today() to run before self.today exists, crashing on restart when the log already exists. The first demo passed because the log was empty—a reminder that developers should test the restart path, not just the happy path.

Token Bucket is a classic rate-limiting algorithm: tokens are continuously replenished into the bucket at a fixed rate, each request consumes one token, and replenishment stops when the bucket is full. Its key characteristic is allowing a degree of "burst"—the bucket's capacity determines how many requests can be overdrawn in a short period, after which requests must wait for new tokens. The article's "12 per minute, burst of 2" means a replenishment rate of 12 tokens per minute with a bucket capacity of 2. Compared to simple fixed-interval rate limiting, the token bucket both smooths long-term rates and accommodates small reasonable bursts, making it the mainstream implementation for API client and gateway rate limiting.

Exponential Backoff is a retry strategy for handling transient failures: after each failure, the wait time grows by a multiple (e.g., 1 second, 2 seconds, 4 seconds, 8 seconds), usually with added random jitter to prevent multiple clients from retrying in sync and causing a "thundering herd." When combined with the server's Retry-After header, the client prioritizes the server-specified wait duration, falling back to exponential backoff only when that header is absent. The article's emphasis that "retrying before Retry-After expires escalates rate limiting into a ban" is precisely the real-world reason this mechanism exists.

robots.txt is a plain-text protocol file at the site root that uses directives like User-agent, Disallow, Allow, and Crawl-delay to tell crawlers which paths are accessible and how access frequency should be controlled. It is not an enforced technical barrier but an industry convention and authorization declaration; obeying it is the basic mark of a "well-behaved" crawler. Python's standard library urllib.robotparser can parse this file and answer "whether a given URL may be crawled." It's important to distinguish: a 404 response for robots.txt means the site has set no rules (allowed by default), while a network failure or timeout means "rules unknown"—the latter should be treated as disallowed under a conservative policy, since you cannot confirm you are permitted.

User-Agent is a string in the HTTP request header that identifies the client's identity and version. An honest User-Agent typically includes the bot's name, purpose description, and a contact URL or email, enabling site operators to reach out rather than immediately ban when traffic anomalies occur. In contrast, a User-Agent disguised as a common browser constitutes detection evasion, running counter to the very purpose of rate limiting.

Real Test: A Simulated Server That "Fights Back"

The demo uses a local simulated server to verify all rules: robots.txt disallows /private and sets Crawl-delay: 1; /profile/* returns a 429 with Retry-After: 2 on every third request; /busy returns a bare 503 on the first request. The client is set to 120 requests per minute so you can observe how crawl delay pushes it down to once per second, and the daily cap is set to 8 for quick exhaustion.

The actual output clearly demonstrates each rule: request intervals are about one second rather than 0.5 seconds, because Crawl-delay overrides the configuration; the 429 waited exactly the 2 seconds the server required; the headerless 503 received about 1.1 seconds of jittered backoff; the /private request never left the local machine; and the last profile request hit the cap mid-retry, because retries are real requests too. The log also records rejected actions, including both robots.txt disallow and reaching the daily cap.

Integrating with the agent is simple: register agent.get as the model's only callable HTTP tool, and treat the Blocked exception as a normal tool result ("not allowed, stop") rather than a bypassable error.

Honest Limitations: Politeness Is Not Permission

The author's stance in this section deserves respect. He points out that politeness is not permission: if a platform's terms prohibit automated access, a slow and clearly labeled bot is still in violation; this tooling reduces collateral damage, not makes prohibited automation acceptable.

Preferring official APIs is the better approach—Meta, X, LinkedIn, and other major platforms all offer APIs with explicit quotas, and informed rate limits beat guessed ones. Logged-in scraping is another category of risk: robots.txt governs crawlers on public pages, but once an agent uses a real person's session, platform account rules take effect, typically prohibiting automated access regardless of pace—and the lost account may belong to your customer.

Human pacing is also not disguise: jitter exists to avoid hammering servers, not to evade detection, and the honest User-Agent is deliberate—if you're thinking about removing it, that itself is a signal that the task should not be automated. Additionally, this solution only covers HTTP, not browsers; if the agent drives a headless browser, each page load triggers dozens of sub-requests, requiring the token bucket to be placed before navigation and clicks, and each operation counted as one action. It's also single-process: five workers means five separate budgets, and cluster scenarios require moving the token bucket and counters to Redis or a database.

The essential difference between official APIs and web scraping lies in authorization and predictability. Official APIs typically require registering an app, obtaining keys, and explicitly specify rate quotas, data usage, and redistribution restrictions in their terms; when exceeded, they return structured error codes and quota headers that clients can use for precise scheduling. Web scraping offers no contractual guarantees: page structures can change at any time, risk-control policies are opaque, and platform terms often cover such behavior with "automated access prohibited." Hence the article's position that "informed rate limits beat guessed ones"—when an official API exists, scraping is not only technically more fragile but also carries higher legal and compliance risk.

Logged-in scraping refers to an agent reusing a real user's session cookies or credentials to access a site. At this point robots.txt no longer applies; platform account rules take over, typically prohibiting automated operations regardless of request pace. Once risk control is triggered, what's damaged is the real user's account rather than an anonymous IP, with consequences potentially including data loss, account suspension, and even broken customer relationships.

Pre-Deployment Checklist

Before an agent touches someone else's site, the author recommends confirming each item: whether an official API exists as an alternative; whether the terms of service permit automated access for this purpose; whether all outbound requests go through the same rate-limited client; whether the User-Agent identifies as a bot and provides valid contact information; whether robots.txt and Crawl-delay are checked; whether Retry-After is honored and backoff applied on 429/503; whether there is a hard daily cap that survives restarts; whether every action can be reconstructed from logs; and if running within a user's logged-in session, whether the user consents and platform rules permit it.

Rate limiting is only half of making an agent safe to deploy; the other half is controlling what the model is allowed to decide. An agent that can only access the necessary tools is safer than a polite but over-privileged one.

One-line Summary

Meta's erroneous ban incident reveals AI agents lack a throttling layer; 124 lines of Python provide seven HTTP guardrail mechanisms.

Paragraph Summary

Starting from the incident where Meta erroneously banned an Instagram account, the article points out that platform risk control looks only at request patterns, not intent, and AI agents inherently lack throttling awareness when calling browser or HTTP tools in loops. The author proposes a 124-line standard-library Python solution integrating seven mechanisms: per-host token buckets, random jitter, Retry-After and exponential backoff, robots.txt checks, honest User-Agent, a daily action cap that survives restarts, and an append-only JSONL audit log. Simulated server tests verified behaviors including Crawl-delay overriding configuration, precise waiting on 429, jittered backoff on headerless 503, and rejected requests being logged. The article also emphasizes that politeness is not permission: automation prohibited by terms remains a violation, official APIs should be preferred, and it honestly notes the solution only covers HTTP and runs single-process, with browser and cluster scenarios requiring separate design.