NVIDIA Dynamo on Kubernetes: Understand AI Inference Deployment in 5 Minutes

From Single-Machine Model Serving to Kubernetes-Native Inference

Many teams start by running a demo on a single model server — simple and direct. But when business requirements become production-grade — needing OpenAI-compatible endpoints, scalable workers, and the ability to change architecture without rewriting glue code — the single-machine approach falls short. In the "Dynamo in 5 Minutes" series, NVIDIA uses Ron's case to break down how to migrate inference services onto Kubernetes.

The core of this migration isn't about stuffing containers into a cluster — it's about making deployment itself a repeatable Kubernetes workflow rather than a pile of custom scripts. The question repeatedly emphasized in the video — "from single model server to Dynamo deployment" — really asks: who handles service discovery, scaling, routing, endpoint exposure, and observability? Dynamo's answer: hand it to the Operator and declarative resources.

The Operator Pattern: Declare Desired State, Let the Cluster Converge

Dynamo adopts the Operator pattern on Kubernetes. Ron's team only needs to describe the desired serving configuration; Kubernetes stores this desired state, and the Operator is responsible for making the actual cluster match it. This gives a very intuitive mental model: define deployment request → apply to cluster → let the Operator create and manage the components → send requests to the OpenAI-compatible endpoint.

The prerequisites aren't complicated: a Kubernetes cluster with access to the required GPUs, and the Dynamo Operator installed via the Dynamo platform Helm chart. From there, teams have two paths to choose from.

Path One: DGDR (Intent-Driven)

The DynamoGraph Deployment Request is the intent-driven path. Teams only need to describe the model, backend, workload, hardware, and optional SLA targets; Dynamo analyzes these inputs and generates a DynamoGraph Deployment (DGD). This path decouples "what I want" from "how to implement it" — ideal for teams that don't want to hand-write topology details.

According to the DGDR flow demonstrated in the video, there are only five steps: define the DGDR (including model, backend, workload, hardware, and optional SLA targets) → apply with kubectl → wait for the Operator to create and reconcile the DGD → send a request to the API endpoint → verify the response actually comes from the Dynamo endpoint. This is the fastest way to get from zero to a working Dynamo endpoint.

SLA Targets and Intent-Driven Deployment

In the context of inference serving, SLA targets typically refer to quantifiable metrics such as latency (e.g., P99 TTFT, end-to-end latency), throughput (tokens/s), or availability. The significance of the intent-driven path is that users don't have to manually decide replica counts, parallelism strategies, or engine parameters — instead, they declare these targets alongside the model and hardware, and the system derives a topology that satisfies the constraints. This allows the same request to be reused across clusters of different scales, and makes it easy to adjust only the targets — rather than rewriting deployment details — when hardware generations change or traffic patterns shift. The trade-off is that the system needs a sufficient tuning knowledge base and search capability, so intent-driven deployment is better suited to common model-and-engine combinations with a high degree of standardization.

Path Two: Using DGD Directly (Pre-Built Recipes)

If a team already has a validated deployment recipe, they can start directly from a DGD manifest. In this case, apply the manifest with kubectl, and the Dynamo Operator reconciles the inference graph into actually running components serving traffic. This path gives teams familiar with the topology more control, at the cost of maintaining the recipe themselves.

Kubernetes Operator and CRD

An Operator is a Kubernetes extension pattern: it encodes operators' domain knowledge into a controller, which continuously watches cluster state and runs a reconcile loop to converge actual state toward the user-declared desired state. This pattern is supported by Custom Resource Definitions (CRDs) — which allow new resource types to be registered in the Kubernetes API, such as the DynamoGraph Deployment (DGD) in this article. After a user submits a CR instance via kubectl, the API Server is only responsible for persisting that declaration; the actual actions — creating Pods, configuring Services, mounting GPUs, exposing endpoints — are performed by the Operator. Unlike Helm, which only does one-time template rendering, an Operator has continuous observation and self-healing capabilities — this is the essential difference between declarative and scripted deployment.

Request Path: Separating a Stable API from an Evolvable Backend

Requests enter through the frontend: clients call the OpenAI-compatible API exposed by the Dynamo frontend, using familiar routes; the frontend acts as the API boundary, routing requests to backend workers; workers run the model engine — SGLang, TensorRT-LLM, or vLLM, depending on the deployment configuration.

The value of this frontend/backend separation is that clients see a stable API while the serving platform behind it can continue to evolve. When starting out, teams can map just one model, one engine, and one worker configuration; when they need to scale, they adjust the deployment rather than rebuilding the entire stack. This is also the key to the "after Dynamo" picture in the video — deployment requests capture intent, the Operator manages the lifecycle, the frontend keeps the interface familiar, and workers are flexibly configured around model and engine choices.

Differences and Choices Among Inference Engines

The three engines mentioned in the article serve different purposes: vLLM is known for PagedAttention and broad model compatibility, with an active ecosystem and a quick learning curve; SGLang focuses on structured generation and prefix caching, with clear advantages in scenarios involving large shared prefixes such as agents and multi-turn conversations; TensorRT-LLM, provided by NVIDIA, pursues peak performance on specific GPUs through compilation optimization and quantization, but requires building for each model. All three can serve as the worker runtime, so by keeping the frontend OpenAI-compatible and switching backends by engine, teams can trade off performance and cost without changing client code.

Migration Advice and Practical Path

For teams migrating a small existing endpoint, the video's advice is: first map the model, engine, and worker settings into the deployment request, start with DGDR to get end-to-end working, verify the OpenAI-compatible endpoint is usable, then adjust the deployment based on capacity needs — rather than refactoring the entire stack.

Dynamo makes Kubernetes deployment more approachable and closer to production-ready by relying on a few clear concepts: manifests, CRDs, the Operator flow, workers, and OpenAI-compatible endpoints. For teams that need to take inference services from experimentation to production, this set of abstractions reduces the repetitive work involved in every architecture change.

FAQ

Does the Dynamo Operator have to be installed via Helm? The video mentions it's typically installed via the Dynamo platform Helm chart, which is the recommended approach.

How should I choose between DGDR and DGD? If you want a working endpoint quickly and don't want to hand-write topology, choose DGDR; if you already have a validated recipe and need fine-grained control over the inference graph, use DGD directly.

Which inference engines do workers support? The video explicitly mentions SGLang, TensorRT-LLM, and vLLM, depending on the deployment configuration.