Cloudflare Responds to AI Cache Vulnerability: Rejected Requests Still Returned Answers

Cloudflare recently published an incident postmortem disclosing a subtle cache isolation flaw in its AI inference service: some overlong requests that should have been rejected not only failed to receive an error, but instead received responses containing data from other users' requests. What makes this incident notable isn't how complex the vulnerability itself was, but how multiple links in the chain of "everything looking normal" stacked on top of each other.

Capacity Gap: A Known Hazard

The postmortem shows that the problematic cache pool could only hold 681208 tokens, while the model's context window was 1048576 tokens. There was a clear capacity gap between the two. Cloudflare acknowledged that this gap was known before the service launched, but at the time it was assumed the engine could handle over-limit situations on its own.

This assumption became the starting point of the entire incident. The engineering team treated "the engine will handle it" as a default premise, without setting up a hard block at the admission layer. When a system relies on an unverified fallback path, problems often don't surface in stress testing—they only emerge under specific input combinations.

Tokens and Context Windows: A token is the basic unit of text processed by a model, roughly corresponding to 0.75 English words or one Chinese character. The context window refers to the maximum number of tokens a model can receive in a single inference, including both the input prompt and generated output. KV Cache is a mechanism in Transformer inference that caches key-value pairs of historical tokens to accelerate decoding, and its VRAM usage is proportional to the number of tokens. When the cache pool capacity is smaller than the context window, overlong requests have nowhere to place their KV state and must rely on the engine's overflow or truncation strategy. This is the technical root of the capacity gap: cache pool size must be planned according to the upper limit of the context window, not estimated based on average request length.

Rejection Flag Is Not a Gate

When a prompt exceeded available space, the engine did mark it as rejected—from a logging perspective, the request had been rejected. But the engine then swapped in a single-token placeholder and continued execution.

What's exposed here is the disconnect between semantics and execution: "rejected" was merely a status flag and did not actually block the subsequent flow. Marking a rejection and actually rejecting are two different things, and the system treated the former as the latter. Cloudflare stated plainly in the postmortem that a flag marked as rejected does not equal a gate.

Rejected Flags and Placeholders: In an inference scheduler, "rejected" is usually just a boolean field on the request object for upper-layer logic to check, and does not automatically interrupt the execution flow. Swapping in a single-token placeholder is a fault-tolerance design intended to keep the decoding loop running with consistent tensor shapes, preventing an entire batch of requests from crashing due to an individual over-limit request. The problem is that this "soft rejection" separates control flow from status flags: the scheduler believes the request has been rejected, while the executor continues advancing with the placeholder. A true gate should terminate the request at the admission layer by throwing an exception or directly returning an error, rather than relying on downstream code to check the flag.

Cache Pages Not Cleared, Decoding Read Stale Data

The more critical link lies in cache management. Cache pages between requests were not cleared, and the decoding step read stale pages that still retained data from previous requests.

This means that even if the current request was "rejected," it could still come into contact with content left in memory pages by the previous user during the decoding stage. This is precisely the source of other customers' request data appearing in AI responses. For multi-tenant inference services, cache reuse is a standard means of improving throughput, but the prerequisite for reuse is a reliable isolation boundary—once the clearing logic is missing, a performance optimization becomes a data leak channel.

Multi-Tenant Inference and Cache Isolation: In multi-tenant inference services, multiple users' requests share the same GPU memory pool. Technologies such as PagedAttention divide the KV Cache into fixed-size pages, allocated and reused on demand to reduce memory fragmentation and improve throughput. The prerequisite for reuse is that pages are cleared or marked invalid before being reallocated; otherwise, a new request may read key-value data left behind by the previous tenant. This type of defect constitutes cross-tenant data leakage, sharing the same root as the classic uninitialized buffer problem. Isolation boundaries must be enforced by the allocator, rather than hoping request logic guarantees "won't read stale pages."

HTTP 200: Making the Anomaly Silent

What the client ultimately received was HTTP 200, not an error code.

This detail determined how difficult the problem was to discover. If over-limit requests returned 4xx, callers would immediately notice and report it; but 200 meant everything "looked normal," and the anomaly was completely silent. Cloudflare stated that a total of 5 responses were affected, all from extremely long prompts, with no evidence of deliberate exploitation. The affected scope was limited, but the chain of issues exposed is worth vigilance: status codes are the sole basis for callers to judge success or failure, and when they are inconsistent with actual execution results, the monitoring system loses its effectiveness.

Status Codes and Observability: HTTP status codes are the first signal for callers to judge whether a request succeeded or failed. 4xx indicates a client error (e.g., an oversized request should return 413), and 5xx indicates a server-side failure. When a service logically fails but returns 200, the caller's retry, alerting, and circuit-breaking mechanisms all become ineffective, and monitoring metrics only reflect "success." For inference services, a clear finish_reason should be included in the response body, and admission rejections should map to 4xx so that anomalies are observable at every layer of the chain. Silent failures often result in longer exposure times than explicit errors.

Remediation Approach and Engineering Lessons

Cloudflare's remediation plan has two parts: first, expand the cache pool to accommodate the full context window; second, directly reject overlong requests at the admission stage. The former eliminates the capacity gap, and the latter ensures rejection occurs before execution rather than during it.

Several lessons of general significance for AI inference services can be drawn from this postmortem: capacity planning cannot rely on the engine's implicit fallback; rejection logic must be implemented as a hard block at the execution layer rather than a status flag; cache reuse must be accompanied by a reliable data clearing mechanism; and errors must be explicitly communicated to callers through status codes. Had any one of these links been done properly, this incident would not have manifested as "silently returning other people's data."

Cloudflare's candid postmortem itself provides a sample: honestly writing decisions like "we knew before launch but assumed we could handle it" into an incident report is more valuable as a reference than simply publishing a remediation plan.

One-line Summary

Cloudflare postmortem on AI inference cache isolation flaw: overlong requests were rejected but still read stale data and silently returned 200.

Paragraph Summary

Cloudflare disclosed a cache isolation incident in its AI inference service: the cache pool capacity of 681208 tokens was smaller than the 1048576-token context window, and the team knew about the gap before launch but assumed the engine could handle it. After overlong prompts were marked as rejected, the engine still swapped in a single-token placeholder and continued execution—the rejection flag did not act as a gate. Meanwhile, cache pages between requests were not cleared, and the decoding stage read pages retaining stale request data, causing responses to be mixed with other customers' content. Clients ultimately received HTTP 200 instead of an error code, making the anomaly completely silent. A total of 5 responses were affected, with no evidence of deliberate exploitation. The remediation is to expand the cache pool to the full context window and directly reject overlong requests at the admission stage.