Dynamo
A distributed inference serving framework introduced by NVIDIA, optimized for GPU utilization and end-to-end latency in multi-node, multi-model and multi-tenant scenarios.
Timeline (last 90 days)
Dynamo is a scheduling and orchestration layer introduced by NVIDIA on top of Triton for large-scale distributed inference scenarios, addressing request routing and resource scheduling challenges in multi-model, multi-node environments.
Google's Bigtable and Amazon's Dynamo are systems papers that demonstrate how to make theoretical trade-offs under engineering constraints.
The eventual consistency concept proposed in the Dynamo paper directly inspired NoSQL databases such as Cassandra and Riak.
All Facts (8)
Dynamo提供分布式KV Cache管理、请求级别的动态路由、GPU资源的细粒度调度等能力
85%UnverifiedDynamo与NVIDIA的TensorRT-LLM推理后端深度集成，同时也支持vLLM等其他推理引擎作为后端
85%UnverifiedDynamo对MoE模型的推理优化包括专家级别的负载均衡和跨GPU的专家并行调度
80%UnverifiedDynamo is a distributed inference framework open-sourced by NVIDIA in early 2025, designed specifically for large-scale LLM serving.
70%UnverifiedDynamo is a scheduling and orchestration layer introduced by NVIDIA on top of Triton for large-scale distributed inference scenarios, addressing request routing and resource scheduling challenges in multi-model, multi-node environments.
50%UnverifiedGoogle's Bigtable and Amazon's Dynamo are systems papers that demonstrate how to make theoretical trade-offs under engineering constraints.
50%UnverifiedThe eventual consistency concept proposed in the Dynamo paper directly inspired NoSQL databases such as Cassandra and Riak.
50%Unverified流式推理与工具执行的流水线并行机制在多工具调用场景下可以将端到端延迟降低30%-50%
50%