Why Rate Limits Can't Stop Model Distillation Attacks

What a Distributed Distillation Attack Actually Looks Like

AI model providers have long relied on rate limiting to prevent abuse — but an overlooked reality is coming into focus: distillation attacks against models have already learned to bypass this mechanism. According to available sources, OpenAI observed 16,000 extraction requests from over 4,000 accounts within just two days — an average of roughly 4 calls per account.

The implications of that number are striking: each individual account looks like a normal user, with request volumes too low to trigger any alert. But zoom out, and those 4,000 accounts converge into a coordinated, large-scale data extraction operation. Rather than hammering the API with a single key, the attackers spread their traffic across a crowd.

Why Rate Limits Fail

The core logic of rate limiting is simple: cap the number of calls per API key or per account. This works well against "single-point, high-frequency abuse" — one key sending a flood of requests is easy to throttle or ban.

The problem is that the shape of distillation attacks has evolved from "individual" to "collective." One line from the original source cuts right to the heart of it: Per-key quotas can't see an actor shaped like a crowd.

When an attacker controls thousands of accounts and each one sends only a handful of requests, any defense built on per-account thresholds is effectively blind. Every account's behavior falls within the distribution of a normal user, giving rate limits no reason to intervene. By adopting a distributed identity, the attacker successfully hides within the noise of legitimate traffic.

The Goal of Distillation Attacks

Model distillation involves querying a target model at scale, collecting input-output pairs, and using them to train a "student model" with comparable capabilities. For an attacker, it's a low-cost way to replicate the capabilities of an expensive frontier model. The question-and-answer data harvested across 16,000 requests is more than enough to form a valuable training corpus.

This explains why providers must take this threat seriously: being distilled means having core capabilities cheaply cloned, posing a direct threat to the commercial investment behind those models.

From a technical implementation standpoint, model distillation attacks typically unfold in several stages. First, the attacker constructs a broad prompt dataset covering diverse task types, systematically probing the target model's outputs across reasoning, coding, creative writing, and Q&A. The collected input-output pairs are then used as supervised training data to fine-tune a smaller, cheaper open-source base model. The end result is a "shadow model" that approximates the target model's capability distribution in specific areas. The key advantage for attackers is that they bear none of the millions of dollars in pretraining compute costs — they only pay for a modest number of API calls and fine-tuning compute. The widespread availability of open-source models like Meta's LLaMA series and Mistral has dramatically lowered the technical bar for launching such attacks: attackers already have a ready-made student model base, which significantly reduces the volume of distillation data required.

Account-Level Controls Are the Answer

The solution the original source points to is clear: Account controls can.

Since the threat takes the form of distributed behavior disguised as a crowd, the defensive lens must shift from the individual key to cross-account behavioral patterns as a whole. In other words, effective detection is no longer about asking "how many times has this key called the API?" — it's about asking "does this cluster of accounts exhibit signs of coordinated extraction?"

Possible defensive dimensions include:

• Account registration and identity verification: Raising the barrier to bulk account creation, making it costly for attackers to cheaply acquire thousands of usable identities.
• Cross-account behavioral correlation analysis: Identifying clusters of accounts whose query patterns are highly similar, temporally synchronized, and systematically covering the boundaries of the model's capabilities.
• Semantic-level anomaly detection: Distillation requests often differ from real user requests in their content distribution — they tend to systematically probe the model rather than solve a specific problem.

What these approaches share is that they focus on the overall shape of behavioral patterns, not the frequency of individual calls.

Cross-account behavioral correlation analysis is no small engineering challenge. Attackers can rotate IP addresses, use different payment methods, and stagger request timing to reduce observable linkage between accounts. More robust detection therefore tends to rely on semantic fingerprints of request content rather than metadata: distillation-type requests exhibit a distinctive "capability probe" character — a high volume of questions about the model's self-awareness (e.g., "what can you do," "explain your reasoning process"), systematic boundary testing across professional domains, and deliberately constructed comparative prompts. This semantic pattern differs statistically from the distribution of real users who have concrete task needs, and can serve as a detection signal. Some providers have also begun explicitly prohibiting distillation-motivated usage in their terms of service, and are using detection findings as legal grounds for account bans — extending the defense beyond pure technical measures into compliance and legal territory.

What This Means for AI Security

This brief observation reveals a necessary shift in how AI platform security must be approached. Traditional API abuse protection rests on the assumption that "frequency equals risk" — but against cost-sensitive, organized distillation attackers, frequency alone is no longer a reliable signal.

The evolution of attacker strategy is a reminder that the boundary of defense must expand from technical metrics to the recognition of behavioral intent. When attack traffic is carefully disguised as a collection of normal users, the only way to see through the disguise is to observe, at a higher level of abstraction, whether this crowd of "users" shares a single underlying objective.

For every provider offering model APIs, this is a signal worth taking seriously. Protecting a model's intellectual property can no longer rely on a single throttling gate.