Deploying NVIDIA Nemotron 3.5 Super VL on AWS EKS and SageMaker
A new architectural reference implementation enables high-performance deployments of the NVIDIA Nemotron 3.5 Super VL multimodal model across Amazon Elastic Kubernetes Service (Amazon EKS) environments.
Radwan AbdulhadiEditor and technology writer
Introduction to NVIDIA Nemotron 3.5 Super VL
NVIDIA Nemotron 3.5 Super VL joins the NVIDIA Nemotron 3 Nano, Nano Omni, Super, Ultra, and Nemotron 3.5 Lightning models already available on AWS. It operates as a 120B-parameter mixture-of-experts model that activates 12 billion parameters per token. By combining advanced reasoning and tool use with comprehensive text, image, document, and video understanding, a single model can effectively serve both text-based and visual tasks.
The open multimodal model is distilled from Nemotron 3 Ultra, with NVIDIA publishing its weights, training datasets, and post-training recipes. This open approach allows developers to customize the architecture for domain-specific workloads and execute them on infrastructure under their direct control. The model utilizes hybrid Mamba-2, MoE, and attention layers alongside a RADIO vision encoder.
Architecture and Performance Capabilities
The hybrid Mamba-Transformer MoE architecture interleaves Mamba-2, attention, and expert layers. Sparse routing activates a fraction of the total parameters, ensuring that the compute cost per token matches a much smaller model while GPU memory retains the full parameter set. Furthermore, Mamba-2 layers maintain a constant-size state during generation, preventing memory from expanding with sequence length in the manner typical of KV caches, which helps keep long-context operations affordable.
Nemotron 3.5 Super VL supports complex agent workflows that seamlessly combine logical reasoning with visual understanding. Use cases span across coding assistance, web research, document intelligence, cyber security threat triage, and video search or incident analysis. The model integrates with major open agent frameworks through official reference implementations and cookbooks.
Deployment on AWS Infrastructure
A detailed technical walkthrough for deploying Nemotron 3.5 Super VL and Nemotron 3 Ultra is hosted via the aws-do-eks project. This workflow covers prerequisites, cluster creation, model weight downloads, deployment, testing, benchmarking, and cleanup procedures. The reference setup is designed to run smoothly on standard infrastructure including Amazon Elastic Kubernetes Service (Amazon EKS) and specialized orchestration platforms.
When utilizing Amazon EKS Auto Mode, AWS manages cluster scaling, GPU configurations, and Elastic Fabric Adapter support on behalf of the user. Kubernetes manifests remain consistent across orchestrator options. Control planes host the core APIs, while data planes separate standard system and test pods on CPU nodes from the resource-heavy model tasks running on GPU nodes.
For specialized high-performance workloads, administrators can also configure deployments using Amazon SageMaker HyperPod EKS. Container images are stored within Amazon Elastic Container Registry, while model weights reside on shared Amazon FSx for Lustre volumes connected via Elastic Fabric Adapter interfaces.
Validation and Benchmarking
NVIDIA Dynamo serves the model either aggregated on a single node or disaggregated, separating prefill and decode phases onto different nodes. For vision-language tasks, image and video encoding stages can run disaggregated on dedicated hardware nodes. These architectures have been thoroughly validated using instances equipped with NVIDIA B200 and NVIDIA H200 GPUs.
Cluster connectivity and network performance are verified using an NVIDIA Collective Communications Library all-reduce micro-benchmark. Observing active traffic across all hardware components confirms proper operational status. Performance metrics such as time to first token, inter-token latency, and token throughput can then be evaluated using the NVIDIA aiperf tool to plan capacity effectively.
Sources
- AWS Machine Learning BlogAccelerate agentic AI with NVIDIA Nemotron models on Amazon EKS and SageMaker HyperPod EKS
Continue chronologically
Related entity coverage
Related stories
Run Multi-Node LLM Inference Using Ray LLM Containers
Learn how to deploy large language models across multiple GPU nodes on Amazon EKS using pre-built Ray LLM Deep Learning Containers.
Adaptive Agentic Intelligence Framework Using Strands Agents
Discover the Adaptive Agentic Intelligence Framework built with Strands Agents to optimize AI workloads using local SLMs and Amazon Bedrock.
AWS Introduces Support for Reflection AI's Beam
Amazon Web Services announces support for Reflection AI's new open-weight agentic coding model, Beam, on Amazon SageMaker.
Scaling MoE Reinforcement Learning on Amazon EKS with EFA
Learn how AWS combines Amazon EKS, EFA, and DeepEP to increase aggregate reinforcement learning rollout throughput by 40% for large-scale MoE training.
