Mooncake
Mooncake is a KV Cache transfer and management system designed for distributed large language model (LLM) inference scenarios. It optimizes inference throughput and latency by efficiently scheduling and transferring KV Cache data across distributed nodes, supports disaggregated deployment of the prefill and decode phases, and features automatic recovery from peer node failures to improve the reliability and resource utilization efficiency of inference clusters.
Timeline (last 90 days)
Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore attempt to extend GPU VRAM capacity using CPU DRAM and SSDs.
Systems such as DistServe, Splitwise, and Mooncake have already implemented disaggregated deployment of prefill and decode.
Splitwise, DistServe, and Mooncake are all representative implementations of prefill-decode (PD) disaggregation.
All Facts (5)
Mooncake通过RDMA网络实现近乎零拷贝的跨节点KV Cache共享
85%UnverifiedMooncake是月之暗面（Moonshot AI）开源的分布式KV Cache传输框架
60%UnverifiedSystems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore attempt to extend GPU VRAM capacity using CPU DRAM and SSDs.
50%UnverifiedSystems such as DistServe, Splitwise, and Mooncake have already implemented disaggregated deployment of prefill and decode.
50%UnverifiedSplitwise, DistServe, and Mooncake are all representative implementations of prefill-decode (PD) disaggregation.
50%