Cloudflare 200 OK Leak: 155 Responses Crossed User Data

Does a 200 response from an AI endpoint really mean everything is fine? Cloudflare's incident gives a definitive no: 155 seemingly normal model responses were actually reading data left in shared GPU memory by other users.

According to a report Cloudflare published on October 26, between August 27 and September 9, 155 responses from the GLM 5.3 model on its Workers AI platform contained other customers' data. The incident was reported by Gnostic researcher Gadi Evron. The report was written by senior engineering managers Kevin Flansburg and Michelle Chen, and was published the same day as SGLang's official statement.

The chain of events: from cache reuse to "false success"

The root cause lies neither in the model nor in its developer z.ai, but in how the inference engine manages shared memory.

KV cache: a "working memory" that isn't automatically wiped

The KV cache is the model's working memory while processing a prompt, stored in reusable shared GPU memory pages. Normally, the engine tracks which page belongs to whom, and overwrites or isolates pages after use. But this time, the window claimed to support roughly a million tokens, while the underlying memory could actually hold only 681,280 tokens. The system knew about this gap and expected to return an error on overlong prompts.

KV cache (Key-Value Cache) is a core optimization mechanism for Transformer-based models during autoregressive generation. Each time the model generates a token, it must compute attention over all previous tokens; without caching, the computation grows quadratically with sequence length. The KV cache stores the Key and Value tensors already produced in each layer's attention computation, so a new token only needs one attention operation against the cached historical state, reducing generation-phase complexity to linear. The cost is that memory usage grows linearly with context length: the longer the sequence, the more layers, and the larger the hidden dimension, the bigger the cache. Inference frameworks typically manage this memory in fixed-size "pages," similar to virtual memory paging in an operating system, and pages can be reused across requests. Because pages are reused, the framework must rigorously track each page's ownership and lifecycle; once ownership records are missing or reuse logic goes wrong, a new request may read the Key/Value left over from a previous request, leaking others' data in the output. This is the mechanistic key to this incident.

A single-token "placeholder" bypassed the error

When SGLang's split-prefill-decode mode encounters an overlong prompt, it replaces the oversized input with a single-token placeholder and continues passing it along. As a result, only the first page is written, while the remaining memory pages retain content left by previous requests—and the model reads this residual data while generating its answer.

split-prefill-decode (prefill/decode disaggregation) is a scheduling mode SGLang designed to improve throughput. Traditional inference performs "reading the entire prompt" (prefill) and "generating token by token" (decode) on the same instance; in disaggregated mode, prefill and decode can be split across different nodes or batches so that compute-intensive and memory-intensive workloads can be optimized separately. This split relies on an explicit assumption: the prefill phase has written the prompt's complete KV state into the shared KV pool, and the decode phase can simply read it. When the prompt length exceeds the KV pool's actual capacity, this assumption no longer holds, and whether the error-handling path is correctly triggered depends on the length-validation logic in each deployment mode. In single-node mode, the check is more direct and returns a length error; in disaggregated deployments, if the length check is bypassed before the actual write, the decode phase reads pages that were never written by this request. Understanding this makes clear why the same model behaves differently across deployment configurations.

Why 200 is more dangerous than 400

The client did not receive a 400 error—it received a 200 and a coherent answer. Cloudflare admits that its tests only checked status codes, so things "looked good." That is precisely what makes this incident so alarming: an error was packaged as success.

In its post-mortem, SGLang stated that this mismatch between context length and KV pool made the rejection path reachable. Single-node mode returns an error normally; only specific deployment modes triggered the problem.

Cloudflare took the model offline within 40 minutes of announcing the incident and restored it the same day after fixing the issue.

Canary text originates from the practice of using canaries to detect poisonous gas in mining; in software testing, it refers to pre-planted marker data that is unique and precisely identifiable. When validating a model API, a typical approach is to embed a randomly generated string (such as a specific UUID or passphrase) in the prompt and then check whether the returned content contains that string verbatim. If the marker is missing from the response, or content appears that does not belong to this request, then the output was not generated entirely from the current input—there may be cache contamination, request cross-talk, or data leakage. Compared with checking HTTP status codes, canary validation directly tests semantic correctness—whether the output really came from this request—and can catch silent errors that status codes cannot reflect. For multi-tenant shared endpoints, it can be combined with output fingerprint comparison and request ID echoing to form a more complete output-integrity verification.

Community debate: should the headline name GLM 5.3?

On Hacker News, some commenters praised Cloudflare for proactively disclosing the problem, while others argued the headline should not point the finger at GLM 5.3—the model itself and its developer were not the cause; the problem lay in the inference engine's deployment configuration. As of now, the researchers' full report has not been released.

An actionable checklist

For teams using SGLang, Cloudflare recommends: run tests in pre-filled decode mode; upgrade to version 0.5.22 or higher; and on each node, verify that KV pool capacity matches the context length configuration.

For any model API, do not rely solely on status codes when validating output—inject known canary text and confirm that the returned content really came from this request. For highly sensitive data, reassess whether shared endpoints are appropriate.

The same habit applies to agent workflows. In Swarm, a completed coding task stops in the needs-review state. Read the diff, run the tests where possible, and then merge. "Done" is just a status, not a check. A 200 only means the server agreed; only the output content can tell you what it actually answered.

It's important to distinguish that an HTTP status code describes the result of request handling at the transport and protocol level, not the correctness of business semantics. A 200 only means the server received and completed a response; it does not guarantee that the response content belongs to the current request, that no cross-tenant data has been mixed in, or that the model output came from this input. Such "protocol success, semantic failure" problems are especially insidious in distributed inference systems, because any state mismatch in intermediate components (scheduler, KV pool, batching) can cause an error to appear as a normal response. In engineering practice, layered validation is typically used to compensate: the protocol layer checks status codes and latency, the semantic layer uses canaries and structural checks, and the data layer confirms boundaries through tenant isolation and audit logs. For security-sensitive workloads, deployment should also ensure that caches are isolated per tenant or zeroed after use, rather than relying on callers to verify each time.

One-line Summary

A Cloudflare incident reveals that model responses with status code 200 may have read residual data from others.

Paragraph Summary

A Cloudflare report disclosed that between August 27 and September 9, 155 responses from the GLM 5.3 model on the Workers AI platform contained other customers' data—all with status code 200. The root cause was neither the model nor its developer z.ai, but the shared-memory management of the inference engine SGLang: the window claimed roughly a million tokens while the actual KV pool held only 681,280 tokens; in split-prefill-decode mode, overlong prompts continued with a single-token placeholder, so only the first page was written while the remaining pages retained residual data from previous requests that the model then read. Cloudflare took the model offline within 40 minutes and fixed it the same day, recommending an upgrade to SGLang 0.5.22 or higher and verification that KV pool capacity matches context length. The core warning is that errors were packaged as success: output validation cannot rely on status codes alone—inject canary text to confirm the content came from this request.