Ray LLM Containers: Deploying Large Models Across GPU Nodes on EKS

Large model inference deployment is moving from single cards to multi-node clusters, and the steep rise in engineering complexity often deters teams. Amazon EKS combined with Ray LLM Deep Learning Containers offers a more pragmatic path: deploy one LLM across two GPU nodes, separate prefill from decode, and expose only a single OpenAI-compatible endpoint to the outside world.

Why Cross-Node Inference Is Needed

When model parameters exceed the memory capacity of a single node, or throughput demand far exceeds what a single node can handle, horizontal scaling is almost the only option. The traditional approach is to assemble the inference stack yourself—scheduling, communication, batching, and health checks built layer by layer—which takes a long time and makes it easy to stumble in failure recovery.

Ray LLM Deep Learning Containers on Amazon EKS change this division of labor. The container already includes a complete inference stack, so the team's focus shifts from "assembling" to "configuring." According to the official description, the core scenario for this solution is: using pre-built containers to distribute a single large language model across two GPU nodes.

The Architectural Logic of Separating Prefill and Decode

LLM inference naturally divides into two stages in terms of compute characteristics. The prefill stage processes the full input prompt, is compute-intensive, and needs to digest prompt length quickly; the decode stage generates output token by token, is memory-bandwidth-intensive, and is extremely sensitive to latency. When both run on the same node, prefill for long prompts crowds out decode resources, causing first-token latency and output smoothness to drag each other down.

The solution in this architecture is to deploy the two types of workloads on different nodes, each doing its own job. Prefill nodes handle "reading the question," while decode nodes handle "answering," with intermediate state such as KV cache exchanged over a high-speed network. This way, each node can tune resource allocation and parallelism strategy for its own workload type, instead of awkwardly accommodating both modes under one configuration.

Practical Impact on Operations

The cost of the separated architecture is that inter-node communication and state transfer must be handled. Fortunately, the container layer already encapsulates this, and on the Kubernetes side it is more about configuration work such as resource allocation, scheduling constraints, and network planning. For teams familiar with K8s, the learning curve mainly lies in understanding the resource profiles of prefill/decode, not in rewriting communication code.

KV cache is a core technical concept in autoregressive inference for large models. During Transformer decoding, each time a new token is generated, the model needs to recompute attention over all previous tokens. If this were recalculated from scratch every time, the cost would grow quadratically with sequence length. KV cache works by caching the Key and Value matrices already obtained in each layer's attention computation, then reusing them directly in subsequent generation to avoid redundant computation. As a result, the per-step computation in the decode stage drops from being related to the square of sequence length to a linear relationship.

The cost of KV cache is memory usage: each concurrent request and each sequence maintains its own independent cache, and usage grows linearly with context length. In a multi-node separated architecture, after the prefill node completes the initial computation, it must transmit the generated KV cache to the decode node over a high-speed network, which is why inter-node communication becomes a key bottleneck. The transmission efficiency of KV cache directly determines whether the separated architecture can deliver performance gains under real workloads.

What a Single OpenAI-Compatible Endpoint Means

Exposing only one OpenAI-compatible interface externally is the most valuable aspect of this solution in real engineering. Upper-layer applications, agent frameworks, and evaluation tools do not need to be aware of whether the backend is single-node or multi-node, nor do they need to distinguish where prefill and decode run.

This means migration cost is minimized: code that originally called the OpenAI API can point to a self-built cluster by changing only the base_url. For teams that need to deploy in a private environment but do not want to refactor client logic, this design saves considerable engineering effort.

At the same time, it raises a point worth weighing—whether the compatibility layer covers all parameters and response fields required by the target application. It is advisable to run a round of interface behavior alignment testing before formal migration, especially for easily overlooked details such as streaming output, stop conditions, and token billing fields.

An OpenAI-compatible interface usually refers to a set of REST endpoints that follow the OpenAI Chat Completions API specification, including paths such as /v1/chat/completions, /v1/completions, and /v1/embeddings, along with the corresponding request body fields (such as messages, temperature, max_tokens) and response structures (such as choices, usage, finish_reason). Many open-source inference frameworks and commercial inference services proactively implement this specification to reduce user integration costs.

Compatibility does not mean complete consistency. Different implementations may differ in how streaming output chunks are split, when finish_reason takes specific values, how stop sequences are handled, and the token counting basis in usage. For applications that rely on precise token billing or strict parsing of stop conditions, these differences can directly lead to inconsistent behavior. Therefore, the interface behavior alignment testing mentioned in the article often focuses not on whether basic conversation works, but on the consistency of these edge behaviors.

When to Choose This Solution

The criteria are not complicated: if your model does not fit into a single node's memory, or single-node throughput has already peaked, and your team is already using EKS, then these pre-built containers can significantly shorten the time from zero to usable.

Conversely, if the model can fit on a single card and traffic volume is not large, introducing multiple nodes will only add scheduling and operational overhead. The benefits of cross-node inference come from scale, not from the architectural sophistication itself.

Another dimension to evaluate is consistency of GPU instance types. Prefill and decode have different preferences for compute power and bandwidth. Although mixing heterogeneous instance types is feasible, it complicates capacity planning. In the early stage, it is advisable to keep node specifications as controllable as possible and minimize tuning variables.

EKS (Amazon Elastic Kubernetes Service) is a managed Kubernetes service provided by AWS. The control plane is maintained by AWS, and users only need to manage node groups or use serverless compute such as Fargate. For multi-node GPU inference scenarios, the value of EKS lies in its mature scheduling, node pool management, elastic scaling, and network plugin ecosystem.

An implicit prerequisite for choosing this solution is that the team already has EKS operational capabilities. If the team has no prior Kubernetes experience, the learning and governance costs of introducing EKS itself may exceed the benefits brought by multi-node inference. The article repeatedly emphasizes that this solution solidifies engineering experience into container images, and its time-saving premise is precisely that underlying capabilities such as scheduling, networking, and storage are already handled by EKS. Without this premise, the convenience brought by container packaging will be offset by the operational complexity of the cluster itself. Therefore, when judging applicability, EKS proficiency should be evaluated as a prerequisite just as important as model scale.

Several Key Configuration Points During Implementation

Image versions and Ray versions need to match. Although pre-built containers save assembly work, version constraints still exist, and when upgrading, all components of the inference stack must be verified in sync.

Network bandwidth is often an underestimated bottleneck. The amount of KV cache transmitted between nodes grows with concurrency and sequence length. If the inter-node link becomes the limiting factor, adding nodes will not buy more throughput.

Observability should be considered during the deployment stage. In multi-node scenarios, breaking down the latency of a single request across multiple components is much harder than in single-node scenarios, and instrumenting in advance saves far more effort than troubleshooting afterward.

This solution essentially solidifies the engineering experience of multi-node inference into container images, letting teams trade configuration for time. For teams that need to run large models on their own infrastructure, this is a pragmatic path worth evaluating.

One-line Summary

With EKS and Ray LLM containers, deploy a large model across two nodes with prefill/decode separation and expose a single OpenAI-compatible endpoint.

Paragraph Summary

The article presents a pragmatic path for multi-node large model inference on Amazon EKS using Ray LLM Deep Learning Containers. The core idea is to split a single LLM across two GPU nodes and separate compute-intensive prefill from bandwidth-intensive decode, optimizing resource allocation for each workload type. Only one OpenAI-compatible interface is exposed externally, so upper-layer applications and agent frameworks do not need to be aware of the backend node topology, resulting in very low migration cost. The solution is suitable when a model exceeds single-node memory or single-node throughput has peaked; small-scale deployments only add overhead. During implementation, key configuration points include matching image and Ray versions, inter-node network bandwidth bottlenecks, and multi-node observability.