Feature/slinky slurm hyperpod eks

[bdaqiq01]

Issue #, if available:

Automate SageMaker HyperPod Slurm Cluster Deployment on EKS

This commit introduces a bash script that automates the deployment of slinky on Amazon a SageMaker HyperPod EKS cluster. Key features include:

• Cluster configuration generation from CloudFormation stack
• FSx for Lustre storage class creation
• AWS Load Balancer Controller installation
• Slinky prerequisites setup (Prometheus, cert-manager, Slurm Operator)
• Dynamic Slurm cluster values configuration based on instance types
• FSx Persistent Volume Claim (PVC) creation and verification
• Slurm cluster deployment with Helm
• Network Load Balancer (NLB) configuration for login access

The script includes error handling, progress tracking, and detailed logging to ensure a smooth deployment process. It's designed to work with both G5 and P5 instance types, dynamically allocating resources based on the GPU count and EFA support of the chosen instances.
However the it is only tested on deployment on the g5 instance types. The resource allocation modification based on the instance type is done on the dynamic_pods_allocation function ( this function is not test due to resource capacity unavailability). Currently this function call is made in this set_slurm_values function. If for g5 instance type it does not work please comment out this line and retry. The commented version should also work for instance type p5 where only one pod is deployed in one node.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[KeitaW]

Hi @bdaqiq01 — this PR was automatically closed by GitHub because the repository history was rewritten as part of a cleanup to reduce clone size (details in #959).

Your fork's branch still references the old (pre-rewrite) history, so it can't be reopened directly. To restore your PR, please rebase your fork onto the new main:

# Update your fork's main branch
cd your-fork
git remote add upstream https://github.com/aws-samples/awsome-distributed-training.git  # if not already set
git fetch upstream
git checkout main
git reset --hard upstream/main
git push --force origin main

# Rebase your feature branch
git checkout feature/slinky-slurm-hyperpod-eks
git rebase main
# Resolve any conflicts if needed
git push --force origin feature/slinky-slurm-hyperpod-eks

# Then open a new PR from your fork

Apologies for the inconvenience! If you run into any issues rebasing, feel free to ask for help.