Artificial Intelligence

Run Multi-Node LLM Inference Using Ray LLM Containers

AWS introduces multi-node Ray LLM Deep Learning Containers designed to streamline the deployment of large language models across multiple GPU nodes behind a single OpenAI-compatible endpoint.

Radwan AbdulhadiEditor and technology writer

Published: October 12, 2026 · Updated: October 12, 2026 · 3 min read

Run Multi-Node LLM Inference Using Ray LLM Containers — Image: AWS Machine Learning Blog

Introduction to Multi-Node Ray LLM Containers

AWS has published a guide on how to deploy one large language model across two GPU nodes on Amazon Elastic Kubernetes Service (Amazon EKS) utilizing multi-node Ray LLM Deep Learning Containers. This architecture places separate prefill and decode nodes behind a single OpenAI-compatible endpoint, making it common for tensor, pipeline, and data parallelism as well as disaggregated inference workloads.

In a previous post, the AWS Machine Learning Blog introduced the Ray Serve DLC to stand up smaller models behind an HTTP endpoint on a single GPU node. Building upon that foundation, the new multi-node approach addresses environments where foundation or open-weight models exceed the capacity of a single GPU, or where workloads require higher throughput and phase separation.

Solving Infrastructure Complexity with Deep Learning Containers

Distributed compute frameworks like Ray allow deployments to span multiple nodes. However, customers previously had to build the underlying dependency chain themselves by selecting compatible libraries for the GPU stack and debugging component drift. AWS Deep Learning Containers (DLCs) solve this by bundling frameworks, dependencies, and GPU stacks into tested, patched Docker images.

The introduction of the Ray LLM DLC extends this pre-built approach to multi-node distributed serving. Maintained and tested by AWS, these purpose-built containers allow users to configure deployments rather than assemble the entire inference stack from scratch.

Architecture diagram: a user request reaches the Ray head on a CPU system node, and two GPU worker nodes run the prefill and decode phases on Amazon EKS — Image related to the report from AWS Machine Learning Blog · Source: AWS Machine Learning Blog

Prefill and Decode Disaggregation Architecture

LLM inference consists of two distinct phases: prefill, which processes the input prompt in parallel to build the key-value cache, and decode, which generates responses token by token. Because prefill is compute-bound and decode is memory-bandwidth-bound, sharing a single GPU causes resource contention where bursty prompts interfere with latency-sensitive generation.

By placing prefill and decode phases on separate GPU nodes, infrastructure teams can scale each phase independently. This disaggregated approach is particularly beneficial for Retrieval Augmented Generation (RAG), document summarization, and agentic applications where maintaining consistent generation speeds is critical.

Container Stack and Network Interconnects

The Ray LLM DLC incorporates vLLM as the inference engine, Ray Serve’s HTTP serving layer, the NVIDIA Inference Transfer Library (NIXL) for fast KV-cache transfers between nodes, and the Elastic Fabric Adapter (EFA) stack for high-bandwidth node-to-node communication. Because the Ray Serve LLM library features a built-in application builder, deployments can be defined using straightforward YAML configurations without writing custom handler classes.

On Amazon EKS, a CPU system node manages system workloads and the KubeRay operator, while two GPU worker nodes handle the prefill and decode phases respectively. The KV cache flows between the workers over NIXL and EFA, keeping network latency low to preserve the performance advantages of disaggregation.