Artificial Intelligence

vLLM Push-Based KV Transfer Boosts TTFT and TPOT

A new push-based NIXL connector architecture for disaggregated large language model inference in vLLM shifts coordination and setup off the critical path, yielding dramatic latency and throughput improvements.

Radwan AbdulhadiEditor and technology writer

Published: October 19, 2026 · Updated: October 19, 2026 · 4 min read

vLLM Push-Based KV Transfer Boosts TTFT and TPOT — Image: AWS Machine Learning Blog

Understanding Prefill-Decode Disaggregation in vLLM

vLLM is an open-source inference and serving engine for large language models. LLM inference splits work into two distinct phases: prefill, which processes the input prompt and builds the key-value cache, and decode, which uses that cache to generate output tokens sequentially. These phases feature fundamentally different compute, memory, and scaling characteristics. Utilizing Prefill-decode (P-D) disaggregation runs these workloads on separate engine instances, allowing each phase to scale and be optimized independently.

While disaggregation can improve resource utilization, throughput, and latency under load, it introduces a new bottleneck. The system must transfer the key-value cache from the prefill instance to the decode instance before token generation can begin, making efficient transfer mechanisms critical to reducing the time-to-first-token latency metric.

Pull-mode timeline: router relay, decode-side allocation, and NIXL READ run serially on the path to first token. — Image related to the report from AWS Machine Learning Blog · Source: AWS Machine Learning Blog

The Limitation of Pull-Based Transfer

One of the traditional ways vLLM performs this transfer is through the NVIDIA Inference Xfer Library connector, which moves key-value blocks over interconnects supporting Remote Direct Memory Access, such as AWS Elastic Fabric Adapter. On supported instance types, AWS EFA supports GPUDirect RDMA and Neuron Direct RDMA, allowing transfers directly between device memories without staging through host memory.

However, the default mode for this connector has historically been pull-based. After the prefill instance finishes computing the prompt cache, a router relays source metadata to the decode instance. The decode instance then allocates destination blocks, prepares the transfer, and initiates a read operation to pull the blocks from the prefill instance. Because these coordination and setup steps occur sequentially after prefill and before decoding starts, they significantly extend the critical path for the first token.

Push-based KV transfer architecture overview — Image related to the report from AWS Machine Learning Blog · Source: AWS Machine Learning Blog

Introducing the Push-Based NIXL Connector

To solve serialization issues and reduce latency, developers introduced a push-based NIXL connector that shifts post-prefill transfer coordination and setup completely off the critical path. If engineers run disaggregated prefill and decode inference using vLLM, this push mode enhances application responsiveness by cutting down time-to-first-token and time-per-output-token metrics, especially under high request rates.

The push design improves performance by overlapping producer and consumer work. The decode instance registers its target blocks in advance while the producer is still computing the key-value cache. The data transfer starts immediately once the producer blocks are ready, eliminating consumer-side ID-discovery round trips from the path to the first token.

Threading model showing the decode and prefill main loops handing KV blocks to a background thread that performs the NIXL transfer off the forward-pass loop — Image related to the report from AWS Machine Learning Blog · Source: AWS Machine Learning Blog

Architecture and Three-Actor Coordination

The push connector successfully coordinates three key actors within the cluster framework. First, the decode instance registers its target blocks up front. Second, the prefill instance matches each request and issues the transfer. Third, a dedicated background thread carries out the key-value write operation entirely off the engine's main forward-pass loop.

During registration, the decode main loop allocates destination blocks and notifies the prefill instance. During the rendezvous phase on the prefill side, the system matches requests by ID, accommodating retried or preempted requests. Finally, the transfer phase issues a direct write that copies the key-value blocks straight into the decoder's registered memory, complete with delivery notifications.

Side-by-side timeline comparing pull and push KV transfer, showing push overlapping decode-side setup with prefill to shorten the path to the first token — Image related to the report from AWS Machine Learning Blog · Source: AWS Machine Learning Blog

Benchmark Results on Amazon EC2 and AWS Trainium

In official benchmarks conducted on a 1P1D deployment using Amazon EC2 P5en instances connected via EFA, the push mode delivered up to 3x better time-to-first-token and up to 30 percent better time-per-output-token compared to standard pull mode when running the Qwen3-32B model, though exact results depend on specific workloads and configurations.

The engineering teams initially built and validated this capability on clusters featuring AWS Trainium before contributing it back to the open-source community as an accelerator-agnostic feature. Performance data confirms that these massive efficiency gains translate seamlessly over to GPU clusters equipped with fast networking fabrics.