Share GPU Clusters Across Teams with Amazon SageMaker HyperPod
Organizations running generative AI operations can now utilize a reference architecture for securely sharing a single Amazon SageMaker HyperPod EKS cluster across multiple internal teams.
Radwan AbdulhadiEditor and technology writer
Addressing Multi-Tenant Challenges in Generative AI
As enterprises increasingly scale their generative AI operations, multiple internal teams often require simultaneous access to expensive GPU clusters. A data science group training large language models, a computer vision team executing inference workloads, and a research unit experimenting with alternative architectures frequently need to tap into the exact same underlying hardware infrastructure. Further details are available from AWS Machine Learning Blog in the original source material.
Without a structured multi-tenant approach, companies often encounter uncontrolled resource consumption, weak team-level isolation, administrative overhead, and significant difficulties in attributing shared GPU expenditures accurately. To resolve these challenges, the AWS Machine Learning Blog detailed a comprehensive reference architecture designed to help organizations efficiently share a single compute cluster.
Leveraging Amazon SageMaker HyperPod and EKS
Amazon SageMaker HyperPod is a specialized service built to simplify the administration of large-scale compute clusters dedicated to generative AI workloads. Orchestrated via Amazon Elastic Kubernetes Service or Slurm, it manages distributed training, interactive development, and inference scaling while automatically handling routine node health monitoring, fault recovery, and cluster lifecycle operations.
Building upon this foundation, the introduced blueprint establishes a multi-tenant environment using an Amazon EKS cluster. It incorporates AWS IAM Identity Center for streamlined authentication, per-team SageMaker AI domains for tailored user experiences, and dedicated Kubernetes namespaces to achieve complete workload isolation between different departments.
Furthermore, the architecture integrates HyperPod Task Governance to maintain fair resource allocation among competing teams, alongside namespace-level cost tracking that grants enterprises clear visibility into individual team spending for accurate chargeback processes.
Centralized Authentication and User Identity Federation
Robust security and authentication form the bedrock of any reliable multi-tenant infrastructure. Centralizing identity management ensures that administrative teams can accurately verify users before granting them access to sensitive cluster resources and computing environments.
Organizations can utilize AWS IAM Identity Center to unify workforce identities across multiple AWS accounts and integrated applications. This service functions as a single source of truth for user identities and group memberships, eliminating the administrative friction of managing scattered user databases.
Additionally, the system allows enterprises to connect an external identity provider such as Microsoft Entra ID, Okta, or Ping Identity. By leveraging these existing corporate identity management systems, companies can seamlessly reuse their current infrastructure without duplicating user profiles.
Dual Access Pathways for Developers and Data Scientists
The multi-tenant reference architecture supports two primary operational paths for users interacting with the shared cluster. In the first pathway, users authenticate through the centralized portal and launch the SageMaker Studio application to access their designated workspace.
Each team accesses its own dedicated SageMaker AI domain, which provides a tailored Amazon SageMaker Studio GUI configured with a specific team execution role. These domains serve as the primary visual interface for running interactive notebooks and submitting machine learning tasks directly toward the underlying cluster.
The second pathway accommodates command-line operations. Users authenticate via the command line interface, obtain temporary credentials mapped to their team's permission set, and submit tasks directly to the EKS cluster using standard Kubernetes command tools.
Strict Isolation and Access Control at the Cluster Boundary
At the boundary of the EKS cluster, advanced access entries securely map specific IAM roles to appropriate Kubernetes permissions. Access entries must be configured for both the Studio execution roles and the CLI console roles provisioned by the identity service, mapping them to managed or custom role-based access control policies.
Because these authorization controls are strictly scoped to each team's designated namespace, users can only view and modify resources residing within their assigned environment, regardless of whether their requests originate from the graphical studio interface or the command line.
This rigorous partitioning extends into the underlying storage infrastructure as well. The environment incorporates POSIX-compliant file systems and object storage buckets organized cleanly into per-team shared directories, individual user home folders, and dedicated S3 storage locations governed by specific execution roles.
Task Governance and Cost Allocation
Beyond isolation and authentication, fairness and resource management remain essential considerations for shared GPU deployments. The architecture incorporates specialized platform layers including observability dashboards for monitoring system health and task governance tools to manage compute quotas and scheduling priorities.
These governance mechanisms prevent any single team from monopolizing the shared cluster resources, ensuring predictable performance for critical workloads across data science, computer vision, and research departments alike.
Finally, namespace-level cost allocation enables financial transparency by tracking resource consumption down to the individual team namespace. This capability allows organizations to execute accurate internal chargebacks, directly tying cloud infrastructure expenditures to the specific business units incurring them.
Sources
- AWS Machine Learning BlogShare GPU clusters across teams with isolation and fairness using Amazon SageMaker HyperPod
Continue chronologically
Related entity coverage
Related stories
Manage SageMaker HyperPod Spaces Directly from Studio
Data scientists and machine learning engineers can now create, configure, start, stop, and open Amazon SageMaker Spaces directly from SageMaker Studio.
AWS Launches Amazon SageMaker HyperPod Inference Gateway
Amazon introduces the SageMaker HyperPod Inference Gateway, a GPU-aware routing add-on for EKS designed to slash inference latency for LLM deployments.
Run Multi-Node LLM Inference Using Ray LLM Containers
Learn how to deploy large language models across multiple GPU nodes on Amazon EKS using pre-built Ray LLM Deep Learning Containers.
Scaling MoE Reinforcement Learning on Amazon EKS with EFA
Learn how AWS combines Amazon EKS, EFA, and DeepEP to increase aggregate reinforcement learning rollout throughput by 40% for large-scale MoE training.
