llm-d

A Kubernetes-native open-source distributed LLM inference stack that supports running across vLLM, SGLang, and heterogeneous TPU/GPU hardware. Its core capabilities include cache-aware routing, agent-aware routing, and multi-tenant flow control.