Deepgram Optimizes Nova-3 Voice AI Scale with AWS Trainium
Deepgram worked alongside the AWS Forward Deployed Engineering team to successfully port its flagship Nova-3 speech-to-text model to AWS Trainium2 hardware, optimizing offline batch inference for voice AI workloads.
Radwan AbdulhadiEditor and technology writer
Bringing Voice AI to AWS Trainium
As voice continues to establish itself as a core modality for artificial intelligence, Deepgram maintains an extensive model portfolio spanning speech-to-text, text-to-speech, and voice agents. Processing quadrillions of audio tokens annually for more than 200,000 developers and 1,400 organizations, the company relies heavily on serving efficiency to determine how much audio its infrastructure can handle within a specific budget. High-volume batch workloads apply intense pressure to these resources, making inference efficiency a primary target for optimization. To address this, Deepgram turned to AWS AI chips to improve performance, collaborating with the AWS Forward Deployed Engineering team to bring its Nova-3 speech-to-text model to AWS Trainium2 hardware for offline batch applications like contact-center analytics and medical transcription. Further details are available from AWS Machine Learning Blog in the original source material.
The development began with standard PyTorch code provided by Deepgram. Over several months, engineering teams ported Nova-3 to AWS Trainium2 using PyTorch reconstruction and performed hardware-specific tuning. The final deployment configuration successfully hosts eight model replicas on a single Amazon EC2 trn2.3xlarge instance, significantly maximizing per-device throughput for throughput-oriented offline batch workloads.
Compiling and Tuning Nova-3 with Native PyTorch
The optimization process utilized TorchNeuron, the native PyTorch integration of the Neuron Compiler, which supports both eager execution and torch.compile. Initial evaluation of Nova-3 in eager mode on Trainium2 identified operations falling back to the CPU before compiling the model with torch.compile, which successfully routed every model operation onto Trainium2 without detected CPU fallbacks. The encoder and decoder components were compiled as separate static-shape graphs, maintaining rigorous numerical accuracy by verifying layer-level output consistency alongside end-to-end Word Error Rate metrics.
Building on this foundation, developers replaced several model layers with fused kernels from the public NKILib library, a collection of pre-built reference kernels written using the Neuron Kernel Interface. These integrated kernels managed operations such as fused QKV projections, fused feed-forward network layers, and depthwise 1D convolution, establishing a baseline for maximum batch throughput.
Kernel-Level Profiling and Tuning
Enabling pre-built kernels marked the beginning of an iterative optimization cycle that involved workload profiling and targeted kernel replacement. Profiling insights helped determine where hand-written kernels could outperform generic options. Throughout this stage, the Neuron Agentic Development toolkit and Neuron Explorer assisted engineering efforts by streamlining profiling analysis and kernel-writing tasks.
Hardware core configurations were similarly refined to maximize efficiency. Because each Trainium2 device includes eight physical NeuronCores typically exposed as four logical NeuronCores, and because each Nova-3 model instance fits comfortably onto a single physical core, the team reduced physical cores per logical core down to one. This configuration enabled the simultaneous hosting of eight model replicas per device while concurrently processing inputs.
Building a Lightweight Rust Runtime
Because Deepgram's primary production serving stack is written in Rust, porting Nova-3 required eliminating Python and PyTorch from the inference serving path. The engineering teams built a lightweight runtime where Rust coordinates model replicas through a shared work-stealing queue. A C layer handles the loading and execution of ahead-of-time-compiled Neuron Executable File Format files directly through the Neuron Runtime API.
Compared to multithreaded and multiprocessing Python runtimes, the custom Rust implementation completely removes interpreter overhead and mitigates Global Interpreter Lock serialization. Because Nova-3 processes a sequence of small decoder calls for each audio chunk, host-side dispatch previously served as a bottleneck at low batch sizes. The Rust runtime addressed this limitation, delivering measurable performance improvements and securing a 7.5 percent throughput gain over the multiprocessing Python baseline at maximum batch sizes.
Throughput and Cost Results
Evaluating performance through audio processed per dollar, the single-device Trainium2 setup running on trn2.3xlarge instances delivered significant cost-efficiency gains. On offline, throughput-oriented batch transcription workloads, the optimized architecture processed between 11 and 70 percent more audio per dollar than the compute options Deepgram previously operated. These architectural efficiencies enable Deepgram to scale operations effectively and pass cost savings along to customers, while extending similar deployment benefits to users self-hosting models within virtual private clouds on Amazon SageMaker AI.
Sources
- AWS Machine Learning BlogHow Deepgram optimized voice AI at scale with AWS Trainium
Continue chronologically
Related entity coverage
Related stories
vLLM Bi-Directional KV Transfer Cuts Multi-Turn TTFT
Learn how bi-directional KV cache transfer in vLLM cuts time to first token for multi-turn agentic workloads on Amazon EC2 instances.
Amazon Launches TorchNeuron to Challenge Nvidia in AI
Amazon Web Services has released TorchNeuron, a new development tool designed to help AI startups run applications on custom Trainium chips.
vLLM Push-Based KV Transfer Boosts TTFT and TPOT
Discover how the new push-based NIXL connector for vLLM disaggregated inference improves TTFT and TPOT performance using AWS infrastructure.
BMW Scales Cloud FinOps with Automated Anomaly Detection
BMW Group utilizes a serverless AWS pipeline to proactively monitor costs across 14,000 cloud accounts, replacing reactive dashboards with AI-driven alerts.
