Cloudflare Remediates KV Cache Exposure in GLM-5.3
Cloudflare has detailed a cross-request KV cache exposure issue affecting the GLM-5.3 Workers AI model and outlined the remediation steps taken alongside SGLang maintainers.
Radwan AbdulhadiEditor and technology writer
Incident Discovery and Timeline
On September 8, 2026, security researcher Gadi Evron from Knostic alerted Cloudflare to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting on August 27, 2026. Cloudflare confirmed that no other model hosted on Workers AI was affected.
Between August 27 and September 9, a small number of customers using the GLM 5.3 Workers AI model received responses containing data from other customers' requests made to the same model shortly before. In total, 155 responses were affected, all involving prompts longer than 681,280 tokens. After excluding internal testing and the researcher's validation, the identified requests matched ordinary long-context use patterns rather than a deliberate exploit.
Following a report submitted through the HackerOne program, Cloudflare reproduced the issue and temporarily took the model offline for disabling the model to secure the infrastructure.
Root Cause: Configuration Mismatch and SGLang Bug
The cross-request KV cache exposure stemmed from two overlapping issues involving the open-source inference engine SGLang. First, Cloudflare deployed GLM 5.3 with a 1,048,576-token context window, while the KV cache pool on prefill GPUs only had room for 681,280 tokens. Although aware of the mismatch, pre-launch long-context tests returned successful responses, leading engineers to expect an error for unsupported prompts.
The second factor was an SGLang bug handling over-length requests in disaggregated prefill and decode mode. When a prompt exceeded the pool, it was internally marked as rejected and replaced with a one-token placeholder. However, instead of aborting the request and returning a client error, the prefill generated a KV cache for the placeholder and transferred it to the decode node.
The decode node assumed prefill overwrote previous states with the full prompt, but the majority of the prior state remained intact. Consequently, the decode node generated responses while attending over stale cache pages holding data from earlier sessions, allowing data leakage in prompts exceeding 681,280 tokens.
Remediation and Configuration Fixes
Before restoring public access, Cloudflare adjusted its GLM 5.3 configuration to match the GLM 5.2 setup. This freed enough GPU memory to expand the prefill KV pool to approximately 2.04 million tokens, roughly twice the 1,048,576-token context length. Valid requests fitting the context window now always fit the pool, bypassing the rejection path entirely.
In addition to expanding the pool, Cloudflare implemented architectural adjustments for high-performance LLMs running on production infrastructure to ensure robust tenant isolation.
Upstream Collaboration with SGLang
Relying solely on pool size proved fragile because changes to quantization, tensor parallelism, CUDA graph sizes, or future versions can alter pool dimensions unexpectedly. To ensure safety, Cloudflare and SGLang maintainers introduced engine patches to retire pending rejections at the disaggregation admission queue before any bootstrap, allocation, or KV transfer occurs.
Cloudflare proposed upstream patches including #38704, which also requires the decode node to verify that the number of transferred tokens matches the reserved count. Maintainers had previously merged #36988 for a related failure path and subsequently incorporated #38935 to reject requests earlier.
More details regarding smaller, faster, safer models and architectural approaches are available in related engineering disclosures published alongside SGLang.
Architecture and KV Cache Management
Workers AI runs GLM 5.3 with SGLang using a disaggregated configuration. A pool of prefill GPUs processes the prompt and produces the model's KV cache, handing it over via remote direct memory access (RDMA) using Mooncake to a pool of decode GPUs generating the final response.
GPU memory assigns fixed-size pages to each request upon admission, releasing them upon completion. Because pages are overwritten by subsequent requests rather than zeroed out between tasks, inference engines must strictly track memory page ownership and lifetimes across both local nodes and remote prefill nodes.
Sources
- Cloudflare BlogCross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang
Continue chronologically
Related entity coverage
Related stories
Cloudflare Launches Clef-omni with Full Multimodality
Cloudflare expands its open-weight decision model family with Clef-omni, featuring full multimodality, faster speeds, and cheaper Clef-flash pricing.
Cloudflare Introduces Clairic System One Extraction Model
Cloudflare has announced a preview of Clairic, a new System One extraction model designed to extract, compute, estimate, and forecast data.
Cloudflare Adds Open European Models and Agnostic AI Defenses
Cloudflare expands its Workers AI platform with Europe's EuroLLM and Switzerland's Apertus while launching model-agnostic cyber defense workshops.
Cloudflare AI Search Launches Into General Availability
Cloudflare's AI Search reaches general availability, introducing native image embeddings, PDF OCR, larger file limits, and new billing terms.
