NIXL
A low-latency KV-cache transfer library provided by NVIDIA, designed for disaggregated LLM inference scenarios. It efficiently transfers KV caches from prefill instances to decode instances after prefill computation is complete.
Timeline (last 90 days)
The key change in v0.30.0rc2 is a fix for notification-only requests in the NIXL transfer component to prevent them from triggering unnecessary receive reports
NIXL (NVIDIA Inference Xfer Library) is a library used to accelerate inference data transfers
In vLLM, NIXL is commonly used to implement cross-node KV cache transfer under a Prefill/Decode disaggregated architecture
NIXL typically leverages high-speed interconnect technologies such as NVLink, InfiniBand, or RDMA at the underlying level to transfer data directly between GPU memory, bypassing the CPU
The validation environment uses eight NVIDIA A40 GPUs, each running one vLLM engine, with KV caches transferred between resource pools via NIXL.
NIXL is a low-latency KV cache transfer library provided by NVIDIA, designed specifically for disaggregated inference scenarios.
NVIDIA Dynamo implements distributed request routing and service discovery based on NATS and etcd, supports efficient inter-GPU data transfer based on NIXL, and is compatible with inference backends such as TensorRT-LLM and vLLM.
All Facts (8)
NIXL是SGLang用于Prefill-Decode分离部署的关键通信组件，负责在Prefill节点和Decode节点之间传输KV Cache状态
85%UnverifiedNIXL typically leverages high-speed interconnect technologies such as NVLink, InfiniBand, or RDMA at the underlying level to transfer data directly between GPU memory, bypassing the CPU
50%UnverifiedThe key change in v0.30.0rc2 is a fix for notification-only requests in the NIXL transfer component to prevent them from triggering unnecessary receive reports
50%UnverifiedNIXL (NVIDIA Inference Xfer Library) is a library used to accelerate inference data transfers
50%UnverifiedIn vLLM, NIXL is commonly used to implement cross-node KV cache transfer under a Prefill/Decode disaggregated architecture
50%UnverifiedThe validation environment uses eight NVIDIA A40 GPUs, each running one vLLM engine, with KV caches transferred between resource pools via NIXL.
50%UnverifiedNIXL is a low-latency KV cache transfer library provided by NVIDIA, designed specifically for disaggregated inference scenarios.
50%UnverifiedNVIDIA Dynamo implements distributed request routing and service discovery based on NATS and etcd, supports efficient inter-GPU data transfer based on NIXL, and is compatible with inference backends such as TensorRT-LLM and vLLM.
50%