NVIDIA Dynamo
A distributed inference framework from NVIDIA designed for efficient serving of large language models across multi-GPU and multi-node clusters. Core capabilities include cross-node KV cache scheduling and reuse, dynamic request routing, and integration with orchestration systems such as Kubernetes.
Timeline (last 90 days)
The industry usually compares LLMD with projects such as NVIDIA's Dynamo and KServe in the same track, with the common focus on how to simultaneously do well in GPU utilization and end-to-end latency under multi-node, multi-model, and multi-tenant conditions.
The official PyTorch blog recently published an article introducing NVIDIA Dynamo's approach to session-aware agentic inference.
NVIDIA Dynamo is a distributed inference framework launched by NVIDIA, designed for efficient serving of large language models on multi-GPU, multi-node clusters.
NVIDIA Dynamo's core capabilities include cross-node scheduling and reuse of KV Cache, dynamic routing of requests, and integration with orchestration systems such as Kubernetes.
Topograph can be combined with NVIDIA Dynamo to route inference requests to the GPU node physically closest to the cached KV data, reducing the communication overhead caused by cache migration.
NVIDIA Dynamo implements distributed request routing and service discovery based on NATS and etcd, supports efficient inter-GPU data transfer based on NIXL, and is compatible with inference backends such as TensorRT-LLM and vLLM.
All Facts (6)
The industry usually compares LLMD with projects such as NVIDIA's Dynamo and KServe in the same track, with the common focus on how to simultaneously do well in GPU utilization and end-to-end latency under multi-node, multi-model, and multi-tenant conditions.
50%UnverifiedNVIDIA Dynamo is a distributed inference framework launched by NVIDIA, designed for efficient serving of large language models on multi-GPU, multi-node clusters.
50%UnverifiedNVIDIA Dynamo's core capabilities include cross-node scheduling and reuse of KV Cache, dynamic routing of requests, and integration with orchestration systems such as Kubernetes.
50%UnverifiedTopograph can be combined with NVIDIA Dynamo to route inference requests to the GPU node physically closest to the cached KV data, reducing the communication overhead caused by cache migration.
50%UnverifiedNVIDIA Dynamo implements distributed request routing and service discovery based on NATS and etcd, supports efficient inter-GPU data transfer based on NIXL, and is compatible with inference backends such as TensorRT-LLM and vLLM.
50%UnverifiedThe official PyTorch blog recently published an article introducing NVIDIA Dynamo's approach to session-aware agentic inference.
50%