EPD Disaggregation: A Deep Dive into Inference Acceleration for Multimodal Models

The Performance Bottleneck of Multimodal Inference

How to efficiently serve inference at scale becomes a core challenge for AI infrastructure when deploying Vision Language Models (VLMs) in production environments. Unlike text-only large language models, multimodal models must first pass image and video inputs through a Vision Encoder stage before entering the Prefill and Decode stages.

The vision encoder is the core component responsible for understanding visual information in multimodal models. Mainstream VLMs typically use a pretrained Vision Transformer (ViT) as the vision encoder, such as CLIP ViT or SigLIP. A ViT divides the input image into fixed-size patches (e.g., 14×14 or 16×16 pixels), linearly projects each patch into a token, and then extracts features through multiple Transformer encoder layers. For a 224×224 image, ViT-L/14 produces 256 visual tokens; with high-resolution images (e.g., 1344×1344) or dynamic resolution strategies, the number of visual tokens can balloon to several thousand. Video inputs require encoding each frame separately, with compute scaling linearly with the number of frames. This enormous variability in computational load is one of the core problems that EPD disaggregation aims to solve.

NVIDIA's Encode-Prefill-Decode (EPD) disaggregation technique decouples the vision encoder from the prefill and decode stages, deploying and scheduling them independently to significantly improve multimodal model serving efficiency.

How EPD Disaggregation Works

Evolving from PD Disaggregation to EPD Disaggregation

Text-only LLM inference already widely adopts Prefill-Decode (PD) disaggregation: the prefill stage is compute-bound, processing all input tokens at once to generate the KV Cache; the decode stage is memory-bandwidth-bound, generating tokens one at a time. Their hardware resource requirements are fundamentally different, and disaggregated deployment enables targeted optimization for each.

The theoretical foundation of PD disaggregation comes from analyzing the distinctly different computational characteristics of the two Transformer inference stages. The prefill stage computes attention matrices in parallel across all input tokens, with computational complexity of O(n²) (where n is the sequence length), making GPU compute (FLOPS) the bottleneck. The decode stage generates only one token at a time and needs to read the entire KV Cache for attention computation, but the actual compute is minimal — GPU memory bandwidth (HBM bandwidth) becomes the bottleneck. In co-located deployments, prefill requests preempt GPU compute from decode requests, causing decode latency jitter (the so-called "prefill preemption" problem). After disaggregation, decode instances can aggregate large batches to improve memory bandwidth utilization, while prefill instances can be configured with higher-compute GPUs. Systems like Splitwise, DistServe, and Mooncake are representative implementations of PD disaggregation.

EPD disaggregation takes this further by additionally separating the Vision Encoder (Encode) stage, which is unique to multimodal models. The vision encoder converts raw inputs like images and videos into embedding vectors, and its computational characteristics differ from both text prefill and decode.

In VLMs, the embedding vectors output by the vision encoder must be aligned with text token embeddings in the same semantic space before being fed into the LLM's Transformer layers for joint reasoning. This alignment is typically accomplished through a projector layer, with common implementations including linear layers, MLPs, Q-Former (as used in BLIP-2), or Perceiver Resampler (as used in Flamingo). The projected visual embeddings are concatenated with text embeddings to form a mixed sequence that serves as input to the LLM. In the EPD disaggregation architecture, the encode stage outputs these projected embeddings, which must be transmitted over the network to the prefill instances. Embedding dimensions are typically 4096 or higher, with each visual token occupying several KB. When the number of visual tokens is large, the transfer overhead is non-trivial.

Resource Characteristics Across the Three Stages

After splitting into three independent stages — encode, prefill, and decode — each stage can be independently scaled and optimized:

• Encode Stage: Processes image/video inputs; compute varies significantly with resolution and frame count; represents a heavy compute workload
• Prefill Stage: Processes the fusion of text tokens and visual embeddings; generates the initial KV Cache
• Decode Stage: Autoregressively generates output tokens; constrained by memory bandwidth

The KV Cache is a critical optimization mechanism in Transformer autoregressive inference. During attention computation, each token must operate on the Key and Value vectors of all preceding tokens. Without caching, generating each new token would require recomputing K and V for all historical tokens, causing compute to grow quadratically with generation length. The KV Cache stores previously computed Key and Value vectors in GPU memory, so each decoding step only needs to compute K and V for the new token and append them to the cache. However, KV Cache memory consumption is substantial — for Llama-70B, a single request with 8K context occupies approximately 1.2GB of GPU memory. In the EPD disaggregation architecture, the KV Cache generated during the prefill stage must be efficiently transferred to decode instances, involving high-speed data movement between GPUs or even across nodes, typically relying on NVLink or RDMA networks.

This decoupling allows each stage to be matched with the most appropriate GPU resource configuration and batching strategy, avoiding bottlenecks from being dragged down by other stages.

When to Use EPD Disaggregation

Three Key Decision Criteria

The applicability of EPD disaggregation is highly dependent on workload characteristics. NVIDIA's analysis identifies the following factors that determine whether it should be adopted:

Proportion of vision encoding overhead. When inputs contain high-resolution images or long videos, the vision encoder consumes significant computational resources. If coupled with prefill and decode on the same instance, resource contention drags down overall throughput. In such cases, EPD disaggregation provides the most obvious benefits.

Diversity of multimodal inputs. In production environments, different requests carry vastly different numbers and sizes of images. With an independent encode stage, dynamic batching and elastic scaling can be applied specifically to the visual workload, improving GPU utilization.

Service Level Objectives (SLOs). For applications sensitive to Time To First Token (TTFT), EPD disaggregation prevents the encode stage from blocking decode, better meeting latency constraints. TTFT is a key metric for measuring inference service responsiveness, defined as the time from request arrival to when the user receives the first generated token. In interactive applications (such as ChatGPT-style conversations or real-time search summaries), TTFT directly affects perceived response speed and is typically required to be within a few hundred milliseconds. SLOs are the quality-of-service targets committed to externally, usually expressed as percentile latencies, such as "P99 TTFT < 500ms." In multimodal inference, if vision encoding, prefill, and decode are co-located on the same instance, a request containing many images may occupy the GPU for several seconds, severely degrading the TTFT for subsequent requests in the queue. EPD disaggregation avoids this "head-of-line blocking" problem by offloading the encode stage to independent instances.

When It's Not Recommended

When vision encoding overhead is minimal (e.g., processing only a single low-resolution image), or when system throughput is already dominated by text processing, the additional communication overhead and architectural complexity of EPD disaggregation may not justify the benefits. The disaggregated approach requires transferring intermediate embedding vectors between instances, which inherently introduces network transfer costs.

NVIDIA Dynamo Implementation

Distributed Inference Framework Support

Deploying EPD disaggregation in practice requires support from an underlying distributed inference framework. NVIDIA Dynamo, a framework designed for large-scale generative AI serving, provides native capabilities for EPD multi-stage decoupling, including efficient inter-stage data transfer, dynamic scheduling, and independent elastic scaling for each stage.

NVIDIA Dynamo is an open-source distributed inference framework released in 2025, purpose-built for large-scale generative AI serving. Its core capabilities include: distributed request routing and service discovery based on NATS and etcd; multi-stage pipeline orchestration where each stage can be independently scaled; a built-in GPU memory-aware intelligent scheduler that dynamically assigns requests based on each instance's KV Cache utilization; and support for efficient GPU-to-GPU data transfer via NIXL (NVIDIA Inference Transfer Library), enabling zero-copy transfer of KV Cache and embedding vectors. Dynamo's multi-stage orchestration capability makes EPD three-stage disaggregation a configuration-level operation rather than an engineering effort requiring inference engine rewrites. It is compatible with inference backends like TensorRT-LLM and vLLM, lowering the migration barrier for teams using different technology stacks.

With Dynamo, developers can configure different quantities and types of GPU resources for the encode, prefill, and decode stages separately, using intelligent routing to efficiently move requests through each stage. This flexible orchestration capability is key to realizing the value of EPD disaggregation in production environments.

Engineering Trade-offs in Practice

In actual deployments, EPD disaggregation introduces new architectural complexity. Teams need to carefully evaluate: the overhead of transferring intermediate embedding vectors, the ratio of instance counts across stages, and scheduling strategies for handling load imbalances. This requires the inference platform to have robust observability and automated tuning capabilities — otherwise, the cost of manual parameter tuning can be prohibitively high.

Key Takeaways

EPD disaggregation represents an important direction in multimodal model inference optimization. As VLMs become prevalent in scenarios like search, content understanding, and intelligent assistants, multimodal inference compute costs are rising rapidly. By decoupling vision encoding, prefill, and decode — three stages with distinctly different resource characteristics — EPD disaggregation enables each stage to be independently optimized and scaled, significantly improving throughput and resource utilization in high visual workload scenarios.

For teams building multimodal inference services, the core decision logic is: first quantify the proportion of vision encoding in overall inference overhead, then evaluate the benefits of disaggregation based on SLO requirements and workload diversity. Only when encoding overhead is sufficiently significant and a mature distributed inference framework is available can EPD disaggregation truly deliver on its performance potential.