Bridging the Compute and Data Gap

Training large artificial intelligence models requires massive graphics processing unit capacity, yet ideal compute resources and training data frequently reside in separate AWS Regions. Accessing datasets across Regions introduces network latency and transfer costs, forcing engineering teams to choose between replicating petabytes of data or absorbing cross-Region latency on every read. To address this trade-off, organizations can pair <a href="https://aws.amazon.com/sagemaker/ai/hyperpod/">Amazon SageMaker HyperPod</a> with <a href="https://qumulo.com/product/aws">Qumulo</a> to keep frontier models current without moving underlying data or sacrificing throughput.

According to findings shared on the <a href="https://aws.amazon.com/blogs/machine-learning/">Artificial Intelligence</a> blog, pairing managed training infrastructure with <a href="https://qumulo.com/product/aws">Cloud Native Qumulo (CNQ)</a> and <a href="https://qumulo.com/product/cloud-data-fabric/">Cloud Data Fabric (CDF)</a> allows training jobs to read datasets stored in separate AWS Regions or on-premises environments. This setup operates without copying data, changing application code, or degrading network performance.

Validation Architecture and Regional Setup

The validation approach evaluated two independent clusters running the same training job. The hub cluster ran in the US East (Ohio) Region, co-located with the dataset, while the spoke cluster operated in the US West (Oregon) Region and read data remotely through CDF over a connection with 60 ms of network latency. In the spoke Region, machine learning developers submitted jobs through an <a href="https://aws.amazon.com/eks/">Amazon Elastic Kubernetes Service (Amazon EKS)</a> orchestrator running on the compute nodes.

Each HyperPod cluster mounted its local CNQ instance over Network File System. The spoke cluster retrieved information from the hub cluster utilizing <a href="https://aws.amazon.com/vpc/">Virtual Private Cloud (VPC)</a> peering, making the entire dataset accessible without requiring direct cross-Region access from individual compute nodes.

Predictive Caching via NeuralCache

To overcome long-haul network barriers, the architecture relies on Qumulo's <a href="https://qumulo.com/product/neural-cache">NeuralCache</a>, which acts as the predictive caching layer inside Cloud Data Fabric. The caching layer observes sequential block reads issued by data loaders and learns specific access patterns. By utilizing an AI model to predict the next 4 KB blocks required by a job, the system pre-caches data locally.

During an initial warmup phase spanning the first 100 to 150 batches, remote clusters experience slightly slower performance while the cache learns. Once warmed, however, the spoke cluster matches the performance of the co-located hub cluster, achieving optimal throughput without extra data orchestration.

Performance and GPU Utilization

Testing demonstrated that remote clusters running outside the data Region achieve optimal utilization comparable to co-located environments. Throughput consistently converges within the opening batches, rendering the one-time warmup duration negligible at scale—representing under one percent of wall-clock time at 10,000 batches and under 0.1 percent past 100,000 batches.

This performance pattern delivers 80 to 90 percent GPU utilization during the initial training phase before quickly converging to 98 to 100 percent utilization. Meanwhile, the underlying storage layer ensures that datasets stored using <a href="https://aws.amazon.com/s3/">Amazon Simple Storage Service (Amazon S3)</a> and managed storage solutions remain easily accessible across distant infrastructure boundaries.

Sources

• AWS Machine Learning BlogMulti-Region training with Amazon SageMaker HyperPod and Qumulo