The KV Cache Leak Incident: The Technical Truth Behind Cloudflare AI's 155 Cross-Conversation Leaks

A Cache Bug That Made AI 'Cross-Talk'

Paste an entire codebase into an AI chat box, hit send, and the answer that comes back fluently discusses someone else's project, someone else's prompt, and even carries the second half of their answer. This isn't science fiction—it's a real incident that happened on Cloudflare Workers AI. According to a postmortem by the YouTube channel Frame by Bytes, Cloudflare officially confirmed that this type of event occurred 155 times between August 27 and September 9, with highly consistent trigger conditions: prompt lengths exceeding 681,000 tokens. No hacker intrusion, no password leak—the culprit was just a cache.

The educational value of this story lies in how it thrusts the most inconspicuous yet most critical component of large model inference—the KV Cache—into the spotlight. Understanding it is the key to understanding why a single instance of "the cache wasn't cleared properly" directly becomes a data leak.

What Exactly Is the KV Cache: The Hotel Front Desk's Notepad

The video uses a vivid analogy: the hotel front desk has a notepad, and each guest is allotted a few pages. They write on them and hand them back, but nobody ever tears out the used pages. The next guest checks in, starts writing from the first page, but when flipping through, also reads the old content on the later pages—which are precisely the notes left by the previous guest.

This notepad is the KV Cache. Every time the model reads in a token, it computes a set of keys and values, stores them in GPU memory, and reuses them when generating subsequent content, avoiding the need to "re-read" the entire conversation each time. The reason it exists is very practical: reading the prompt is one of the slowest steps in inference. The creator ran a controlled experiment on his own laptop with a small model: after reading in a long document, keeping the cache and asking a follow-up question yielded an order-of-magnitude improvement in response speed compared to discarding the cache and recomputing. Keep the cache, keep the speed; lose the cache, and it's like reading from scratch every time.

Tokens and the Context Window

To understand the KV Cache's capacity problem, we first need to clarify two concepts: tokens and the context window. A token is the basic unit of text the model processes; an English word may be split into one or more tokens, and a Chinese character typically corresponds to one or two tokens. The model cannot "see" infinitely long text at once—it is trained with an agreed maximum processing length, the context window. When input exceeds this length, the model either truncates or throws an error.

The key point is that the context window is the model's logical upper limit, while the KV Cache's capacity is the hardware's physical upper limit. The relationship is similar to "the manual says maximum load is 1 ton" versus "this truck's cargo bed actually has only 0.7 tons of volume"—specifications and implementation do not automatically align. The root cause of the incident in this article is precisely that these two numbers were set separately by human decision: the model window was set to one million tokens, while the page pool held only about 681,000 tokens. Understanding this is what makes it clear why "the prompt is legal but doesn't fit" is not a contradiction but a configuration-level oversight.

The Failure Chain: The Pool Can't Hold It, But the Pages Aren't Cleared

All requests share the same block of GPU memory. Model weights take up the bulk, and the remainder is divided into "pages," forming the so-called page pool. According to the video, the pool capacity on Cloudflare machines was about 681,000 tokens, but the model was told it could handle a context window of one million tokens. Thus a fatal gap emerged: the prompt could fit into the model's window but not into the cache's pool. And pages were never zeroed out when returned—the next request simply overwrote them.

At this scale, a single machine could no longer keep up, so requests were split across two stages: Prefill and Decode. Prefill reads in the prompt and fills pages; Decode generates the answer token by token, with pages passed between them over the network. The router tells Decode in advance how many pages to reserve.

The incident occurred in this handoff protocol. When Byte's ultra-long prompt arrived, Prefill determined it exceeded the pool capacity, marked it as rejected, and replaced the prompt with a single-token placeholder. On a single machine, this would simply return a 400 error. But across machines, Decode only asked "is the task complete?" and never asked "was the task rejected?" So Decode reserved eight pages as originally planned; Prefill sent only one placeholder page, yet Decode read all eight—seven of which belonged to the previous user, carrying their prompt and half-written answer. The model processed this residual content into a coherent reply and even attached a 200 OK status code.

The technical lesson here is quite clear: a 200 status code never equals correct content. When a system passes state between distributed components, any unchecked "implicit assumption" can become an entry point for a vulnerability.

The Division of Labor Between Prefill and Decode

Large model inference is typically divided into two phases. The Prefill phase reads in the entire user prompt at once, computes the keys and values for all tokens in parallel, and writes them to the KV Cache; this phase is compute-intensive, with time growing linearly with input length. The Decode phase generates the answer token by token, reading the KV cache of all previous tokens for each new token, making it memory-access-intensive.

Splitting the two phases across different machines allows Prefill machines to focus on long inputs while Decode machines continuously produce output, improving overall throughput. But this split introduces network transmission and state synchronization problems: the Decode machine must know in advance which pages to reserve space for, and this information depends entirely on what Prefill communicates. Once the communicated content is incomplete—such as saying "complete" but not "rejected"—Decode reads memory according to a wrong expectation, which is precisely the architectural cause of the 155 cross-conversation incidents.

The Three-Part Fix: Pool Expansion, Entry-Point Interception, and Cleanup Still in Draft

Cloudflare's fix came in three steps. First, expand the page pool beyond the model's context window, fundamentally eliminating the mismatch of "fits in the window but not in the pool." Second, the inference engine SGLang rejects overlong prompts at the entry point, intercepting before any pages are reserved; this change shipped with version 0.5.22. Third, zeroing out pages upon return—but as of the video's publication, this item remained in draft status.

The third item has been slow to land for a reason that is blunt and somewhat cruel: zeroing out incurs a speed cost, and speed is the entire reason the cache exists. Security and performance form a real tension here—this is not negligence by some team, but a structural trade-off across the entire technical approach.

What Is SGLang

SGLang, mentioned in the article, is an open-source large model inference engine focused on high-throughput, low-latency server-side deployment. One of its core features is paged KV Cache management, an approach borrowed from operating system virtual memory paging. It divides GPU memory into fixed-size pages, allocating them on demand to different requests, thereby supporting long contexts and concurrent batching.

Precisely because SGLang manages page allocation and reclamation, the fix landed there: version 0.5.22 added length validation at the entry point, so that overlong prompts are rejected before any pages are reserved. This also illustrates an engineering principle—cache lifecycle management should be centralized in the layer responsible for allocation, rather than scattered across implicit agreements among multiple components. Understanding SGLang's role helps assess why this type of fix was chosen to land at that layer, and why zeroing out has been slow to advance because it would touch the inference hot path.

Lessons for Developers: A Cache Is Someone Else's Memory

The advice Frame by Bytes offers at the end applies to everyone who writes code: whether you're operating the KV Cache in GPU memory or any cache in your own application, remember its essence—it is memory that someone else just used. Before reading a page, confirm who owns it.

From an engineering perspective, there are three layers worth chewing over in this incident. The first is lifecycle management: cache allocation, reuse, and reclamation must have clear ownership boundaries, rather than relying on the luck that "it'll probably be overwritten." The second is cross-component contracts: the interface between Prefill and Decode passed only "complete" and omitted "rejected." This kind of missing information is harmless in a monolithic architecture but directly causes data cross-talk in a distributed one. The third is vigilance about performance: any cleanup step omitted in the name of speed will come back to collect its debt in the form of a security incident at some point.

For ordinary users, this incident also provides a basis for judgment: when you use a hosted AI service, whether your prompt length and content might touch the capacity boundaries of underlying resources is not an entirely imperceptible risk. A service provider's capacity planning and isolation mechanisms directly determine whether your data could appear in a stranger's chat box.

Frequently Asked Questions

What content can a KV Cache leak expose?

According to the postmortem of this incident, what leaked was the previous user's prompt and the partially generated answer, both of which remained in GPU memory as cache pages and were misread by the next request.

Why was it only triggered by ultra-long prompts?

Because only when the prompt exceeds the page pool capacity does Prefill trigger the rejection process and substitute a placeholder, and the cross-machine handoff protocol happened not to check this rejection status, causing Decode to read pages beyond what was actually sent.

How should ordinary users view this type of risk?

The root cause of such incidents lies in server-side architecture, and users cannot directly prevent them. A feasible approach is to pay attention to service providers' incident disclosures and remediation progress, and to be cautious about uploading sensitive content.

One-line Summary

A KV Cache that was never zeroed plus a flawed handoff protocol caused 155 cross-conversation leaks at Cloudflare.

Paragraph Summary

The article reviews 155 cross-conversation leak incidents on Cloudflare Workers AI between August 27 and September 9. The trigger condition was prompts exceeding 681,000 tokens; the root cause was that the page pool capacity was smaller than the model's one-million-token context window, and cache pages were never zeroed out when returned. When an overlong prompt arrived, Prefill determined it exceeded capacity and substituted a single-token placeholder, but the cross-machine handoff protocol only checked whether the task was complete, not whether it was rejected. Decode therefore read eight pages as originally planned, seven of which belonged to the previous user, causing their prompt and half-finished answer to be processed into a coherent reply and returned. The fix included pool expansion and entry-point interception in SGLang 0.5.22, while page zeroing remained in draft due to its performance cost. The article draws engineering lessons about cache ownership, cross-component contracts, and performance trade-offs.