vLLM Bi-Directional KV Transfer Cuts Multi-Turn TTFT
A new bi-directional key-value cache transfer mechanism in vLLM helps reduce time to first token latency for multi-turn agentic workloads by allowing prefill and decode nodes to reuse previously computed conversation context across turns.
Radwan AbdulhadiEditor and technology writer
The Challenge of Multi-Turn Agentic Workloads
Multi-turn agentic applications continually build on accumulated conversation history as an agent reasons, calls tools, and incorporates prior results. During each new turn, the client resends the full conversation history, which causes the growing context to be processed repeatedly and drives up latency. Standard Prefill-decode (P-D) disaggregation splits each request across specialized infrastructure where prefill nodes process prompts to build their key-value cache before decode nodes retrieve those blocks to generate responses.
Conventional disaggregated serving only moves cache entries in one direction from prefill to decode nodes. This one-way data flow creates inefficiencies because decode nodes hold cache entries for both prompts and newly generated tokens after a response is finished. When the next turn arrives, prefill nodes cannot retrieve those entries, forcing them to recompute projections for previous response tokens and consume accelerator capacity as conversations grow.
Architecture of Bi-Directional KV Transfer
To address these operational inefficiencies, bi-directional key-value cache transfer enables decode nodes to return previously computed cache blocks back to prefill nodes on subsequent conversation turns. This capability prevents the recomputation of shared context on prefill nodes based on a configurable recompute threshold. Coordination relies on a KV-cache-aware router that maintains an in-memory mapping from conversation IDs to decode transfer parameters.
Developers can explore the available reference materials and framework code to understand how control flows through the proxy. An example proxy is provided in the official repository to assist teams looking to implement similar disaggregated serving patterns in their own production environments.
Enhancements to the NIXL KV Connector
Engineers extended the NVIDIA Inference Xfer Library KV connector so that prefill nodes can consume remote cache blocks similarly to how decode nodes consume prefill-provided blocks. Because prefill and decode nodes carry architectural differences—with prefill nodes generating complete prompt representations and decode nodes serving previously cached blocks—each node retains its distinct role for computing remote and local metadata while utilizing a common read mechanism.
Under this bi-directional mode, decode instances retain blocks for a specified time-to-live duration rather than freeing them immediately after generation finishes. If a subsequent turn arrives before the expiration window, the prefill node reuses the cached information, whereas late requests fall back to normal prefill operations.
Performance and Benchmark Results
In multi-turn benchmarks conducted on a single-prefill, single-decode deployment utilizing Amazon Elastic Compute Cloud instances, bi-directional transfer significantly reduced the time to first token. The evaluation demonstrated a reduction of up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8 workloads.
These hardware deployments leverage low-latency remote direct memory access networks. On supported instance types, the infrastructure moves cache blocks directly between device memory across nodes to optimize communication efficiency.
Network Integration and Reproduction
The underlying architecture connects servers using AWS Elastic Fabric Adapter, which supports advanced remote direct memory access protocols like GPUDirect RDMA and Neuron Direct RDMA for fast block transfers across graphics processing units and custom accelerators.
The feature was initially developed and validated on AWS Trainium instance clusters before being contributed to vLLM as an accelerator-agnostic capability. Teams interested in validating these metrics independently can consult the detailed GitHub reproduction guide provided in the project documentation.
Sources
- AWS Machine Learning BlogImprove TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
Continue chronologically
Related entity coverage
Related stories
vLLM Push-Based KV Transfer Boosts TTFT and TPOT
Discover how the new push-based NIXL connector for vLLM disaggregated inference improves TTFT and TPOT performance using AWS infrastructure.
Deepgram Optimizes Nova-3 Voice AI Scale with AWS Trainium
Deepgram collaborated with AWS to port its Nova-3 speech-to-text model to AWS Trainium2, boosting batch transcription throughput and efficiency.
BMW Scales Cloud FinOps with Automated Anomaly Detection
BMW Group utilizes a serverless AWS pipeline to proactively monitor costs across 14,000 cloud accounts, replacing reactive dashboards with AI-driven alerts.
Deploying NVIDIA Nemotron 3.5 Super VL on AWS EKS and SageMaker
Learn how to deploy the NVIDIA Nemotron 3.5 Super VL mixture-of-experts model on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS.
