NVIDIA Dynamo KV Routing in Practice: Smart Routing Configuration Cuts TTFT by 40%

NVIDIA Dynamo is an open-source, high-throughput, low-latency inference framework purpose-built for deploying large-scale generative AI and inference models across multi-node, multi-GPU environments. Its core component, the KV Router, intelligently distributes inference requests by evaluating the computational cost of each worker. According to official data, KV-aware routing can reduce Time to First Token (TTFT) by up to 40% and boost overall throughput by 25-30% compared to traditional round-robin load balancing.

This article, based on a hands-on guide published by Vultr, provides a complete walkthrough of Dynamo's smart routing configuration workflow, parameter tuning strategies, and validation methods, covering both aggregated and disaggregated serving modes.

How KV-Aware Routing Works

Traditional load balancing strategies (such as round-robin or random routing) distribute requests evenly across workers without considering each worker's cache state. KV-aware routing, by contrast, determines which workers already have relevant KV blocks cached and directs requests to the nodes with higher cache hit rates, thereby reducing the computational load during the prefill phase.

Cost Function and Routing Decisions

The KV Router makes routing decisions based on a unified cost function that balances two key factors:

Prefill cost measures the number of tokens a worker needs to process from scratch. When a worker has KV blocks cached that match the request's input prefix, those blocks can be reused, reducing the prefill computation:

Prefill blocks = (Total input tokens - Cached tokens) / Block size

Decode cost tracks the number of active blocks on each worker to estimate decoding capacity. Workers with fewer active sequences naturally have more headroom to accept new requests.

The two are combined into the final cost formula:

cost = overlap_score_weight × prefill_blocks + decode_blocks

Here, overlap_score_weight controls the trade-off between TTFT optimization and load distribution. The default value is 1.0, and a lower cost indicates a better routing choice.

The Actual Selection Process

Consider 3 workers with overlap_score_weight = 1.0. The routing decision plays out as follows:

Worker	Cached Blocks	Prefill Blocks	Decode Blocks	Total Cost	Selected
Worker 1	2	8	10	18	No
Worker 2	5	5	5	10	Yes
Worker 3	8	2	9	11	No

Worker 2 strikes the best balance between cache reuse and available capacity. This mechanism delivers the most significant gains in scenarios with long input prompts (over 2000 tokens), similar prefixes across requests, and multiple workers each maintaining independent KV caches.

The Cost of KV Cache and the Prefill Phase: In large language model inference, the Transformer's self-attention mechanism must store Key and Value vectors for every processed token, and this storage is the KV Cache. When a request arrives, the model must first run a forward pass over all input tokens to populate the KV Cache — this stage is called prefill — before generating token by token in the decode phase. The floating-point operations of prefill grow roughly quadratically with input length, making it the primary source of TTFT (Time to First Token). If a worker has previously processed a prompt with the same prefix as the current request, the corresponding blocks already exist in its KV Cache and can be reused directly, skipping redundant computation.

Block-Granularity Cache Management: To facilitate scheduling and cross-worker transfer, the industry (e.g., vLLM's PagedAttention) divides the KV Cache into fixed-size blocks, typically 16 or 64 tokens. Tracking cache hits at the block level lets the router quickly estimate the number of reusable tokens and also makes it feasible to move cache across a disaggregated architecture via NIXL/RDMA — only the needed blocks are transferred rather than the entire sequence. The "Cached tokens" and "Block size" in the formulas above are based on this granularity.

The Fundamental Trade-off Between TTFT and Throughput: TTFT optimization seeks to let each request reuse cache as quickly as possible to reduce waiting, which concentrates traffic on a few "hot" workers; throughput optimization seeks to spread load across all GPUs to maximize overall utilization. These goals are opposed, so the router needs an adjustable weight parameter to slide between them — which is why DYN_KV_OVERLAP_SCORE_WEIGHT is the core tuning knob.

Environment Preparation and Infrastructure Setup

Hardware and Software Prerequisites

The following requirements must be met before configuring smart routing:

• Aggregated serving requires 2 GPUs, while disaggregated serving requires 4
• A GPU server with the NVIDIA Container Toolkit configured
• Docker Engine and Docker Compose
• A Hugging Face account and access token (for restricted models such as Llama)
• Understanding of basic LLM inference concepts (prefill/decode phases, KV cache)

Cloning the Repository and Container Image

First, clone the Dynamo repository and check out a stable version:

$ git clone https://github.com/ai-dynamo/dynamo.git
$ cd dynamo
$ git checkout release/0.9.0

Next, verify the CUDA version. Use nvidia-smi to check the CUDA version supported by the driver, then pull the vLLM runtime container with the corresponding tag from NGC. For CUDA 13.x users:

$ docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13

For CUDA 12.x users, use the tag without a suffix:

$ docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0

A CUDA version mismatch will lead to incompatible GPU kernels, so be sure the host driver and container runtime are consistent.

Hugging Face Cache Permissions

The container runs as UID 1000 and needs write permission to download model weights:

$ mkdir -p container/.cache/huggingface
$ sudo chown -R 1000:1000 container/.cache/huggingface
$ sudo chmod -R 775 container/.cache/huggingface

Improper permission settings will directly cause worker initialization to fail.

Starting Infrastructure Services

Dynamo's KV Router depends on two infrastructure services: etcd for worker discovery and registration, and NATS for real-time propagation of KV cache events (block creation and eviction). Start them with a single Docker Compose command:

$ docker compose -f deploy/docker-compose.yml up -d
$ docker compose -f deploy/docker-compose.yml ps

Confirm that both the deploy-etcd-server-1 and deploy-nats-server-1 containers are running.

Why etcd and NATS Are Both Essential: A router in a distributed inference system must know "which workers are currently online and what addresses they listen on" — etcd handles this responsibility. etcd is a strongly consistent distributed key-value store, commonly used as a service discovery and configuration center. In Dynamo, each worker registers its metadata with etcd on startup and periodically renews its lease; when a worker crashes or goes offline, the lease expires and it is automatically removed, preventing the router from sending requests to dead nodes.

The Real-Time Event Channel That NATS Handles: If etcd maintains a "static membership list," NATS is responsible for the "dynamic cache state stream." vLLM workers generate events when creating or evicting KV blocks, and Dynamo's bridge component converts these events into NATS messages. The router subscribes to them and can learn within milliseconds which blocks have been added or invalidated on a given worker. Without this pipeline, the router could only "guess" cache distribution from its own routing history, with significantly reduced accuracy. NATS also provides JetStream persistence for syncing persistent state such as cache blocks across frontend replicas — the foundation for fault-tolerant multi-replica deployments.

Port and Network Assumptions for Local Deployment: The Docker Compose template exposes etcd's port 2379 and NATS's port 4222 on the host by default. All workers and the frontend access these two services via localhost or the container network, so no additional configuration is typically needed in single-machine multi-GPU scenarios. If you scale to multiple nodes, however, you must change the etcd and NATS addresses to reachable cluster addresses and ensure the firewall allows the corresponding ports — otherwise worker registration and event subscription will fail silently.

Core Routing Parameters Explained

The KV Router controls routing behavior, cache tracking, and worker selection logic through environment variables. All of the following parameters can be set on the Dynamo frontend via environment variables or command-line arguments.

Key Routing Parameters

Variable	Description	Default	Recommended
DYN_ROUTER_MODE	Routing strategy mode	round_robin	kv
DYN_ROUTER_TEMPERATURE	Controls routing randomness via softmax sampling	0.0	0.1
DYN_KV_OVERLAP_SCORE_WEIGHT	Weight for prefix cache overlap	1.0	1.5
DYN_KV_CACHE_BLOCK_SIZE	KV cache block size (number of tokens)	Depends on backend	16
DYN_KV_EVENTS	Enable real-time KV event tracking	true	true

The DYN_KV_OVERLAP_SCORE_WEIGHT is the most critical tuning parameter — higher values favor TTFT optimization, while lower values favor load balancing. DYN_KV_CACHE_BLOCK_SIZE must match the backend's block size (e.g., vLLM), otherwise cost calculations will be incorrect.

Worker Capacity Thresholds

These two parameters define when a worker is considered "busy" and excluded from routing:

• DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD: Threshold based on KV cache block utilization, recommended at 0.85 (between 0.0 and 1.0)
• DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD: Threshold based on the absolute number of prefill tokens, recommended at 10000

The two use OR logic: if either threshold is breached, the worker is marked as busy. Thresholds can also be updated dynamically at runtime via the /busy_threshold API without restarting the frontend.

Parameter Tuning Strategies

TTFT-Optimized (Low-Latency Scenarios):

• Increase DYN_KV_OVERLAP_SCORE_WEIGHT to 1.5-2.0 to prioritize cache hits
• Lower the busy thresholds: DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.70, DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=500
• Keep DYN_ROUTER_TEMPERATURE between 0.0-0.1 to ensure deterministic selection

Throughput-Optimized (High-Load Scenarios):

• Lower DYN_KV_OVERLAP_SCORE_WEIGHT to 0.5-1.0 to balance cache hits and load
• Raise the busy thresholds: DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.95, DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=2000
• Increase DYN_ROUTER_TEMPERATURE to 0.3-0.5 to improve load distribution

Aggregated Serving Configuration in Practice

Aggregated serving places the prefill and decode phases on the same worker. KV-aware routing optimizes request distribution based on cached KV blocks, delivering performance gains even when a worker handles both phases simultaneously.

Deployment Script

Create an aggregated deployment script inside the container that launches two vLLM workers (on GPU 0 and 1) with KV event publishing enabled. The frontend is configured for KV-aware routing:

#!/bin/bash
source ~/kv_routing.env
MODEL="${1:-nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1}"

python -m dynamo.frontend --router-reset-states &
sleep 10

# Worker 1 - GPU 0 (ZMQ port 5557)
DYN_SYSTEM_PORT=${DYN_SYSTEM_PORT1:-8081} \
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --trust-remote-code \
    --connector none \
    --enable-prefix-caching \
    --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5557"}' &

The key design choices include: --connector none indicates aggregated mode; --enable-prefix-caching enables prefix cache reuse; each worker is configured with its own ZMQ port and system port to avoid conflicts.

KV Event Stream Architecture

vLLM natively publishes KV cache events via ZMQ, and Dynamo's bridge component automatically converts these events into NATS messages for the router to consume:

vLLM Worker 1 (ZMQ:5557) ──┐
                            ├──→ Dynamo Bridge ──→ NATS Server ──→ KV Router
vLLM Worker 2 (ZMQ:5558) ──┘

This architecture gives the router a unified view of cache state across all workers while preserving vLLM's native event publishing mechanism.

Verifying Routing Effectiveness

After sending the first request, the frontend logs show the cost calculation for each worker:

Formula for worker_id=...892 with 0 cached blocks: 5.188 = 1.5 * 2.125 + 2.000
Selected worker: worker_id=...892, logit: 5.188, cached blocks: 0

On a cold start, neither worker has cached blocks, so the costs are identical (5.188), and the router selects one via temperature sampling.

After sending a second request with a similar prefix, cache reuse takes effect:

Formula for worker_id=...890 with 0 cached blocks: 5.375 = 1.5 * 2.250 + 2.000
Formula for worker_id=...892 with 1 cached blocks: 3.875 = 1.5 * 1.250 + 2.000
Selected worker: worker_id=...892, logit: 3.875, cached blocks: 1

The cost drops from 5.375 to 3.875, and the prefill computation shrinks from 2.250 blocks to 1.250 blocks, translating directly into faster first-token responses.

Disaggregated Serving Configuration in Practice

Disaggregated serving splits the prefill and decode phases into separate worker pools. The KV Router optimizes both prefill worker selection (based on cache hits) and decode worker allocation (based on load), coordinating KV cache transfer between phases via NIXL.

Deployment Architecture

A disaggregated configuration requires 4 GPUs: GPUs 0-1 as decode workers and GPUs 2-3 as prefill workers. The key difference is the use of --connector nixl to enable KV cache transfer via NIXL, with prefill workers using --is-prefill-worker and decode workers using --is-decode-worker.

# Decode Worker - GPU 0
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --connector nixl \
    --is-decode-worker \
    --enable-prefix-caching \
    --trust-remote-code &

# Prefill Worker - GPU 2 (ZMQ port 5559)
CUDA_VISIBLE_DEVICES=2 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --connector nixl \
    --is-prefill-worker \
    --enable-prefix-caching \
    --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5559"}' &

Dual Routing Decisions

In disaggregated mode, the frontend makes two rounds of routing decisions. Prefill routing selects workers based on cache overlap, while decode routing allocates based on load.

On the first request, no prefill worker has a cache, so the costs are identical (4.125). In the decode routing formula, the prefill block weight is 0.0 — decode workers don't compute prefill; they receive KV cache via NIXL:

Formula: 2.000 = 0.0 * 2.750 + 2.000

When a second request with a similar prefix arrives, prefill routing shows the effect of cache reuse:

Formula for worker ...930 with 1 cached blocks: 2.250 = 1.5 * 1.500 + 0.000
Formula for worker ...934 with 0 cached blocks: 3.750 = 1.5 * 2.500 + 0.000

The cache hit reduces the cost from 3.750 to 2.250, once again validating the effectiveness of KV-aware routing for TTFT optimization.

Verifying NIXL Transfer

The worker logs output the result of the NIXL compatibility check:

INFO nixl_connector._nixl_handshake: NIXL compatibility check passed

NIXL uses RDMA to achieve low-latency, high-bandwidth KV cache block transfer between prefill and decode workers, which is the key enabler of disaggregated architecture performance.

What Is NIXL: NIXL (NVIDIA Inference Xfer Library) is a point-to-point data transfer library designed by NVIDIA for inference scenarios, abstracting the available high-speed interconnects (NVLink, InfiniBand RDMA, RoCE, and even TCP fallback) into a unified interface. In a disaggregated architecture, the KV Cache blocks computed by prefill workers must be moved to decode workers to continue generation, and this transfer would severely slow the pipeline if done via ordinary memory copies. NIXL enables data to move directly between GPU memory, bypassing the CPU and system memory, minimizing transfer overhead.

RDMA and Zero-Copy: RDMA (Remote Direct Memory Access) allows a machine's network card to read and write another machine's memory directly, without involving the other machine's CPU and without repeated copies between kernel space and user space. NIXL uses RDMA to write a KV block from prefill GPU memory into decode GPU memory, with end-to-end latency typically in the tens of microseconds. This is why disaggregated deployments have hardware requirements — if the cluster lacks an RDMA network (such as InfiniBand or RoCE-capable Ethernet), transfer degrades to TCP, potentially negating the benefits of disaggregation.

What the Handshake Check Means: The NIXL compatibility check passed in the logs indicates that the two workers have agreed on the transfer protocol, memory registration method, and device capabilities. If this check fails, common causes include: mismatched CUDA driver versions between the two ends, missing corresponding RDMA user-space libraries (such as libibverbs), or containers not running with --privileged / properly mounted devices. In this case, workers usually don't crash, but cache transfer fails, causing missing data in the decode phase — manifesting as garbled output or request errors.

Multi-Backend Support and Unified Configuration

The KV Router's environment configuration is consistent across the three major inference backends: vLLM, SGLang, and TensorRT-LLM. The same environment variables can be reused directly, with only backend-specific adjustments needed for block size and KV event publishing method.

Block Size Differences

Backend	Typical Block Size	Configuration
SGLang	16	DYN_KV_CACHE_BLOCK_SIZE=16
TensorRT-LLM	64 or 128	DYN_KV_CACHE_BLOCK_SIZE=64
vLLM	16	DYN_KV_CACHE_BLOCK_SIZE=16

A block size mismatch between the router and workers leads to incorrect cost calculations and faulty routing decisions, so be sure to verify the backend configuration.

Event Publishing Methods Compared

SGLang uses the NATS URL format: --kv-events-config nats://localhost:4222. TensorRT-LLM requires just one flag: --publish-events-and-metrics. vLLM uses a JSON configuration format to specify the ZMQ endpoint.

Prediction Mode

When the backend does not support KV event publishing, you can switch to prediction mode via --no-kv-events. The router predicts cache state based on its own routing decisions, using a TTL expiration mechanism (default 120 seconds). Prediction mode is less accurate than real-time events but eliminates the NATS dependency and lowers infrastructure overhead.

The Trade-off Between Real-Time Event Mode and Prediction Mode: Real-time event mode (the default) gives the router a "god's-eye view" — it knows exactly which blocks each worker currently has cached, making routing decisions the most accurate. The cost is that NATS must be running and a continuous event stream processed, creating a dependency on infrastructure and network stability. Prediction mode (--no-kv-events) forgoes the external event source, and the router infers each worker's cache state from the requests it has dispatched: it assumes a request sent to a worker leaves corresponding blocks on that worker and records the creation time of those blocks.

TTL Expiration Mechanism: Since there are no eviction event notifications, prediction mode must rely on timeouts to simulate block invalidation. The default 120-second TTL means the router considers a block valid for 120 seconds after creation, removing it from its local view once it expires. This value needs to match the backend's actual cache eviction policy: setting it too long causes the router to send requests to workers that have actually evicted the block, resulting in false hits without reduced prefill; setting it too short underestimates cache hit rates and degrades toward round-robin behavior.

Applicable Scenarios: Prediction mode suits small- to medium-scale deployments where the backend itself doesn't support KV event publishing, or where you want to simplify infrastructure (no NATS deployment), and it's also commonly used for quick validation. However, it's less accurate for high-concurrency scenarios with highly repetitive request prefixes and frequent cache evictions, since the deviation between the inferred view and the real state is amplified when cache state changes rapidly. In production, real-time event mode is recommended, paired with monitoring of cache hit rate metrics.

Production Monitoring and Scaling

Deploying Grafana and Prometheus lets you visualize KV cache routing performance metrics, tracking TTFT improvements, cache hit rates, and worker selection patterns.

When multiple frontend replicas are needed for fault tolerance and horizontal scaling, enable the --router-replica-sync flag. Each replica maintains its own view of active block state (transient state); replicas exchange routing decisions via NATS core messaging, while cache block state (persistent state) is automatically synchronized via JetStream. New replicas start with zero active blocks but converge quickly through synchronization, ensuring consistent routing decisions.

Follow-up optimization directions for production deployments include: tuning DYN_KV_OVERLAP_SCORE_WEIGHT and DYN_ROUTER_TEMPERATURE based on real traffic patterns; building Grafana monitoring dashboards to continuously track TTFT, cache hit rates, and worker load; and testing SGLang or TensorRT-LLM backends to compare cache efficiency.

One-line Summary

NVIDIA Dynamo's KV-aware routing directs requests to workers with higher cache hit rates via a cost function, reducing TTFT by 30-40%.

Paragraph Summary

This article systematically covers the configuration and practice of the KV Router in the NVIDIA Dynamo inference framework. KV-aware routing evaluates each worker based on a cost function (the weighted sum of prefill blocks and decode blocks), directing requests to the node with the highest cache reuse rate — reducing TTFT by 30-40% and boosting throughput by 25-30% compared to round-robin load balancing. The article details environment preparation (etcd, NATS, and container images), core parameter tuning (the trade-off between DYN_KV_OVERLAP_SCORE_WEIGHT and DYN_ROUTER_TEMPERATURE), the differences between aggregated and disaggregated deployment modes, and unified configuration across vLLM, SGLang, and TensorRT-LLM backends along with the prediction mode fallback.