RadixAttention
RadixAttention is an attention mechanism optimization technique proposed in the SGLang inference framework that uses a radix tree data structure for unified KV cache management, enabling automatic reuse of key-value caches across requests. Its core capability involves identifying shared prefixes in scenarios such as multi-turn dialogue and batch inference and reusing corresponding cached values, thereby reducing redundant computation, lowering GPU memory consumption, and improving large language model inference throughput.
Core Facts
SGLang's core innovation is the RadixAttention technology, which achieves automatic KV Cache reuse through a prefix tree structure.
SGLang achieves efficient KV Cache reuse through innovations such as RadixAttention, and can deliver several-fold throughput improvements when handling complex multi-turn dialogue and Agent Loop scenarios.
Timeline (last 90 days)
sglang is a high-performance serving framework optimized for large language model inference that supports continuous batching and RadixAttention, offering throughput advantages over Ollama.
RadixAttention can identify prefix segments shared across different requests and directly reuse cached computation results, reducing redundant computation in scenarios with fixed system prompts, multi-turn conversations, or batch document Q&A, thereby improving throughput and reducing time-to-first-token (TTFT).
SGLang's RadixAttention organizes the KV of prompts and generated results into a RadixTree, with common prefixes at the upper levels; paper data shows that the additional overhead is below 0.3% for workloads without prefix reuse.
In concurrent request scenarios sharing the same system instructions, RadixAttention can reduce prefill-stage computation by more than 80%.
RadixAttention manages shared prefixes through a Radix Tree structure to reduce redundant computation.
In concurrent request scenarios sharing system instructions, RadixAttention can reduce computation in the prefill stage by more than 80%.
RadixAttention automatically reuses KV Cache across multiple requests through a radix tree structure, managing shared prefixes to reduce redundant computation and improve inference throughput.
SGLang's RadixAttention technology can automatically reuse KV Cache across multiple requests, managing shared prefixes through a radix tree structure to reduce redundant computation.
SGLang is developed by LMSYS, focuses on structured output and complex workflows, and introduces the RadixAttention mechanism, which can automatically detect and reuse the common prefix of prompts
SGLang is a high-performance large model inference and serving framework developed by the UC Berkeley LMSYS team, which optimizes KV Cache reuse efficiency through technologies such as RadixAttention.
1 more timeline events
All Facts (15)
SGLang's core innovation is the RadixAttention technology, which achieves automatic KV Cache reuse through a prefix tree structure.
85%VerifiedSGLang achieves efficient KV Cache reuse through innovations such as RadixAttention, and can deliver several-fold throughput improvements when handling complex multi-turn dialogue and Agent Loop scenarios.
75%UnverifiedSGLang的RadixAttention对于共享系统提示（system prompt）的场景可带来数倍吞吐量提升
75%UnverifiedSGLang's RadixAttention technology can automatically reuse KV Cache across multiple requests, managing shared prefixes through a radix tree structure to reduce redundant computation.
70%UnverifiedSGLang 引入了 RadixAttention 技术，在某些基准测试中推理速度可比 vLLM 等主流推理框架提升 2-5 倍
70%Unverifiedsglang is a high-performance serving framework optimized for large language model inference that supports continuous batching and RadixAttention, offering throughput advantages over Ollama.
50%UnverifiedRadixAttention can identify prefix segments shared across different requests and directly reuse cached computation results, reducing redundant computation in scenarios with fixed system prompts, multi-turn conversations, or batch document Q&A, thereby improving throughput and reducing time-to-first-token (TTFT).
50%UnverifiedSGLang's RadixAttention organizes the KV of prompts and generated results into a RadixTree, with common prefixes at the upper levels; paper data shows that the additional overhead is below 0.3% for workloads without prefix reuse.
50%UnverifiedRadixAttention manages shared prefixes through a Radix Tree structure to reduce redundant computation.
50%UnverifiedIn concurrent request scenarios sharing the same system instructions, RadixAttention can reduce prefill-stage computation by more than 80%.
50%UnverifiedIn concurrent request scenarios sharing system instructions, RadixAttention can reduce computation in the prefill stage by more than 80%.
50%UnverifiedRadixAttention automatically reuses KV Cache across multiple requests through a radix tree structure, managing shared prefixes to reduce redundant computation and improve inference throughput.
50%UnverifiedSGLang is developed by LMSYS, focuses on structured output and complex workflows, and introduces the RadixAttention mechanism, which can automatically detect and reuse the common prefix of prompts
50%UnverifiedSGLang is a high-performance large model inference and serving framework developed by the UC Berkeley LMSYS team, which optimizes KV Cache reuse efficiency through technologies such as RadixAttention.
50%UnverifiedRadixAttention 引入全局共享 KV Cache 池，通过引用计数和 LRU 淘汰策略管理缓存生命周期，并在 Radix Tree 中执行最长前缀匹配
50%