fix: drop defunct cloud_reg_addrs; document real mechanism (resume.py) (#7)

Resolves the gen6 blocker from live validation, and corrects a documentation
error that predated gen6.

Research (against SchedMD source): cloud_reg_addrs is NOT a documented slurm.conf
option — absent from slurm.conf.5 in 23.11, 24.05, 25.05, and 25.11. It was a no-op
warning on <=24.05 and is a hard error ('defunct') on 25.11. Burst-node addressing
is actually done by the AWS plugin's resume.py, which runs
'scontrol update nodename=<n> nodeaddr=<ip> nodehostname=<h>' after the EC2 instance
gets its IP — the documented default cloud path (see cloud_dns in slurm.conf.5).

Config (all generations that carried it):
- gen3, gen5, gen6 slurm.conf.tpl: SlurmctldParameters=idle_on_node_suspend
  (dropped cloud_reg_addrs). gen1/gen2 never had it. Unblocks gen6 on Slurm 25.11;
  a no-op change for gen3/gen5 since the plugin already set NodeAddr.

Docs (corrected the inaccurate 'key improvement' narrative):
- slurm-gen3-deep-dive.md: rewrote the section to credit resume.py; fixed the
  comparison table, query/watch examples, and troubleshooting steps.
- generations.md: rewrote the Gen 3 'key architectural improvement' block and the
  Gen 5/6 mentions; fixed the two arc diagrams and the choose-a-gen table; replaced
  the now-invalid cloud_reg_addrs demo example.
- README, sa-guide, support-matrix: dropped cloud_reg_addrs as a listed feature.

All remaining cloud_reg_addrs mentions repo-wide are explicit 'defunct/removed/was
incorrect' correction notes. terraform validate passes on gen3/5/6; doc links clean.

This does not require re-validating gen3/gen5 on live AWS: the parameter was inert
there (plugin owns addressing), so removing it cannot change their behavior.

Author: scttfrdmn

README.md

@@ -50,7 +50,7 @@ burstlab/
 │   ├── architecture.md              # Network, EFS, NAT, IAM, and security design
 │   ├── slurm-gen1-deep-dive.md      # Every slurm.conf directive explained (Gen 1)
 │   ├── slurm-gen2-deep-dive.md      # Gen 2 config changes from Gen 1
-│   ├── slurm-gen3-deep-dive.md      # Gen 3 config changes, cloud_reg_addrs
+│   ├── slurm-gen3-deep-dive.md      # Gen 3 config changes, cgroup v2
 │   ├── plugin-v2-setup.md           # Plugin v2 setup: configs, debugging, IAM
 │   ├── sa-guide.md                  # How to use BurstLab with customers
 │   └── workloads/                   # Workloads overlay: data staging demos
@@ -205,20 +205,20 @@ Each is an independently deployable cluster matching a specific customer environ
 |---|---|---|---|---|
 | **Gen 1** | Rocky 8 | 22.05.x | Python 3.6 boto3 shim; cgroup v1; FSx ✅ | RHEL/Rocky 8, Slurm 22.x — largest installed base |
 | **Gen 2** | Rocky 9 | 23.11.x | Python 3.9 native; cgroup v2; FSx ✅ | RHEL/Rocky 9, Slurm 23.x |
-| **Gen 3** | Rocky 10 | 24.05.x | `cloud_reg_addrs`; cgroup v2 only; Ed25519; [FSx requires burstlab-lustre](https://github.com/scttfrdmn/burstlab-lustre) | RHEL/Rocky 10, Slurm 24.x; greenfield |
+| **Gen 3** | Rocky 10 | 24.05.x | cgroup v2 only; Ed25519; [FSx requires burstlab-lustre](https://github.com/scttfrdmn/burstlab-lustre) | RHEL/Rocky 10, Slurm 24.x; greenfield |
 
 ### Ubuntu Track
 
 | Generation | OS | Slurm | Key Features | When to Use |
 |---|---|---|---|---|
 | **Gen 4** | Ubuntu 22.04 | 23.11.x | apt/AppArmor; Python 3.10; cgroup v2; [FSx requires burstlab-lustre](https://github.com/scttfrdmn/burstlab-lustre) | Ubuntu 22.04, Slurm 23.x; academic/cloud-native/NVIDIA |
-| **Gen 5** | Ubuntu 24.04 | 24.05.x | apt/AppArmor; Python 3.12; `cloud_reg_addrs`; [FSx requires burstlab-lustre](https://github.com/scttfrdmn/burstlab-lustre) | Ubuntu 24.04, Slurm 24.x; latest LTS |
+| **Gen 5** | Ubuntu 24.04 | 24.05.x | apt/AppArmor; Python 3.12; [FSx requires burstlab-lustre](https://github.com/scttfrdmn/burstlab-lustre) | Ubuntu 24.04, Slurm 24.x; latest LTS |
 
 ### Managed-AWS-Slurm Track
 
 | Generation | OS | Slurm | Key Features | When to Use |
 |---|---|---|---|---|
-| **Gen 6** | Amazon Linux 2023 | 25.11.x | `cloud_reg_addrs`; cgroup v2 only; matches ParallelCluster/PCS; **authored, not yet validated** | Migrating to/from AWS ParallelCluster 3.15.x or AWS PCS |
+| **Gen 6** | Amazon Linux 2023 | 25.11.x | cgroup v2 only; matches ParallelCluster/PCS; **authored, not yet validated** | Migrating to/from AWS ParallelCluster 3.15.x or AWS PCS |
 
 **Start with Gen 1 for RHEL/Rocky customers** or **Gen 4 for Ubuntu customers** unless you know
 the specific OS and Slurm version. Most HPC teams struggling with cloud bursting today are on
@@ -342,7 +342,7 @@ BurstLab eliminates the "can we even get it working" phase. The Terraform and co
 | [architecture.md](docs/architecture.md) | SAs, technical customers | Network, EFS, NAT, IAM deep dive |
 | [slurm-gen1-deep-dive.md](docs/slurm-gen1-deep-dive.md) | SAs, HPC admins | Every slurm.conf directive for Gen 1 |
 | [slurm-gen2-deep-dive.md](docs/slurm-gen2-deep-dive.md) | SAs, HPC admins | Gen 2 config changes from Gen 1 |
-| [slurm-gen3-deep-dive.md](docs/slurm-gen3-deep-dive.md) | SAs, HPC admins | Gen 3 config changes, `cloud_reg_addrs` |
+| [slurm-gen3-deep-dive.md](docs/slurm-gen3-deep-dive.md) | SAs, HPC admins | Gen 3 config changes, cgroup v2 |
 | [plugin-v2-setup.md](docs/plugin-v2-setup.md) | SAs, HPC admins | Plugin v2 setup, configs, debugging |
 | [sa-guide.md](docs/sa-guide.md) | SAs | How to run a customer demo |
 | [workloads/overview.md](docs/workloads/overview.md) | SAs | Workloads overlay: scenario guide, storage tiers |

configs/gen3-slurm2405-rocky10/slurm.conf.tpl

@@ -7,10 +7,9 @@
 # on every node via EFS.
 #
 # Gen 3 additions vs Gen 2 (23.11/Rocky 9):
-#   - cloud_reg_addrs: burst nodes register with their actual EC2 IP address.
-#     slurmctld accepts any address from a CLOUD node without pre-configuration.
-#     This eliminates the "SLURM_NODENAME must match configured NodeAddr" failure
-#     mode when EC2 assigns a different private IP than expected.
+#   - Burst node addresses are set by the AWS plugin's resume.py (scontrol update
+#     nodeaddr=...), not by a slurm.conf parameter. See the Controller Parameters
+#     note below on why cloud_reg_addrs is NOT used.
 #   - TaskPlugin=task/cgroup: Rocky 10 uses cgroup v2 exclusively;
 #     task/affinity is removed (deprecated in 24.x for pure-cgroup configs).
 #   - cgroup.conf uses CgroupPlugin=cgroup/v2 (see separate file).
@@ -41,11 +40,13 @@ SelectTypeParameters=CR_Core_Memory
 
 # --- Controller Parameters ---------------------------------------------------
 # idle_on_node_suspend: powered-down cloud nodes show as IDLE (not IDLE~) in sinfo.
-# cloud_reg_addrs: cloud nodes register with their actual EC2 IP. slurmctld
-#   records the address at registration time and uses it for subsequent comms.
-#   This is the key Gen 3 improvement: burst nodes no longer need NodeAddr
-#   pre-configured or matching a specific IP.
-SlurmctldParameters=idle_on_node_suspend,cloud_reg_addrs
+# NOTE: cloud_reg_addrs was removed. It is not a documented slurm.conf option (absent
+#   from slurm.conf.5 through 24.05) and is redundant here: burst node addresses are
+#   set by the AWS plugin's resume.py, which runs `scontrol update nodename=<n>
+#   nodeaddr=<ip> nodehostname=<h>` once the EC2 instance has an IP — the documented
+#   default cloud-registration path (see the `cloud_dns` option). On 24.05 the
+#   parameter only logged a warning; Slurm 25.11 rejects it as defunct.
+SlurmctldParameters=idle_on_node_suspend
 
 # --- Accounting --------------------------------------------------------------
 AccountingStorageType=accounting_storage/slurmdbd

configs/gen5-slurm2405-ubuntu2404/slurm.conf.tpl

@@ -6,12 +6,11 @@
 # at deploy time via templatefile(). Written to /opt/slurm/etc/slurm.conf
 # on every node via EFS.
 #
-# Gen 3 additions vs Gen 2 (23.11/Rocky 9):
-#   - cloud_reg_addrs: burst nodes register with their actual EC2 IP address.
-#     slurmctld accepts any address from a CLOUD node without pre-configuration.
-#     This eliminates the "SLURM_NODENAME must match configured NodeAddr" failure
-#     mode when EC2 assigns a different private IP than expected.
-#   - TaskPlugin=task/cgroup: Rocky 10 uses cgroup v2 exclusively;
+# Gen 5 (Slurm 24.05 on Ubuntu 24.04):
+#   - Burst node addresses are set by the AWS plugin's resume.py (scontrol update
+#     nodeaddr=...), not by a slurm.conf parameter. See the Controller Parameters
+#     note below on why cloud_reg_addrs is NOT used.
+#   - TaskPlugin=task/cgroup: Ubuntu 24.04 uses cgroup v2 exclusively;
 #     task/affinity is removed (deprecated in 24.x for pure-cgroup configs).
 #   - cgroup.conf uses CgroupPlugin=cgroup/v2 (see separate file).
 # =============================================================================
@@ -41,11 +40,13 @@ SelectTypeParameters=CR_Core_Memory
 
 # --- Controller Parameters ---------------------------------------------------
 # idle_on_node_suspend: powered-down cloud nodes show as IDLE (not IDLE~) in sinfo.
-# cloud_reg_addrs: cloud nodes register with their actual EC2 IP. slurmctld
-#   records the address at registration time and uses it for subsequent comms.
-#   This is the key Gen 3 improvement: burst nodes no longer need NodeAddr
-#   pre-configured or matching a specific IP.
-SlurmctldParameters=idle_on_node_suspend,cloud_reg_addrs
+# NOTE: cloud_reg_addrs was removed. It is not a documented slurm.conf option (absent
+#   from slurm.conf.5 through 24.05) and is redundant here: burst node addresses are
+#   set by the AWS plugin's resume.py, which runs `scontrol update nodename=<n>
+#   nodeaddr=<ip> nodehostname=<h>` once the EC2 instance has an IP — the documented
+#   default cloud-registration path (see the `cloud_dns` option). On 24.05 the
+#   parameter only logged a warning; Slurm 25.11 rejects it as defunct.
+SlurmctldParameters=idle_on_node_suspend
 
 # --- Accounting --------------------------------------------------------------
 AccountingStorageType=accounting_storage/slurmdbd

configs/gen6-slurm2511-alinux2023/slurm.conf.tpl

@@ -6,11 +6,15 @@
 # at deploy time via templatefile(). Written to /opt/slurm/etc/slurm.conf
 # on every node via EFS.
 #
-# Gen 6 carries the Slurm 24.05+ feature set (same as Gen 3/5) onto Slurm 25.11:
-#   - cloud_reg_addrs: burst nodes register with their actual EC2 IP address.
-#     slurmctld accepts any address from a CLOUD node without pre-configuration.
-#     This eliminates the "SLURM_NODENAME must match configured NodeAddr" failure
-#     mode when EC2 assigns a different private IP than expected.
+# Gen 6 notes (Slurm 25.11 on AL2023):
+#   - NO cloud_reg_addrs. Slurm 25.11 makes that SlurmctldParameters option a hard
+#     error ("defunct"). It was never a documented option (absent from slurm.conf.5
+#     through 24.05) and was always redundant here: the AWS plugin's resume.py pushes
+#     each burst node's address in explicitly via `scontrol update nodename=<n>
+#     nodeaddr=<ip> nodehostname=<h>` once the EC2 instance has an IP. That is exactly
+#     the default cloud-node registration path Slurm documents (see `cloud_dns`), so
+#     dropping the parameter changes nothing operationally — the plugin still sets the
+#     address. (gen3/gen5 carry it too but only warn on 24.05; 25.11 rejects it.)
 #   - TaskPlugin=task/cgroup: AL2023 uses cgroup v2 exclusively;
 #     task/affinity is removed (deprecated in 24.x for pure-cgroup configs).
 #   - cgroup.conf uses CgroupPlugin=cgroup/v2 (see separate file).
@@ -41,11 +45,10 @@ SelectTypeParameters=CR_Core_Memory
 
 # --- Controller Parameters ---------------------------------------------------
 # idle_on_node_suspend: powered-down cloud nodes show as IDLE (not IDLE~) in sinfo.
-# cloud_reg_addrs: cloud nodes register with their actual EC2 IP. slurmctld
-#   records the address at registration time and uses it for subsequent comms.
-#   This is the key Gen 6 improvement: burst nodes no longer need NodeAddr
-#   pre-configured or matching a specific IP.
-SlurmctldParameters=idle_on_node_suspend,cloud_reg_addrs
+#   (Still valid in Slurm 25.11.)
+# NOTE: cloud_reg_addrs is intentionally absent — it is defunct in Slurm 25.11 and
+#   redundant with the AWS plugin (resume.py pushes NodeAddr in). See header.
+SlurmctldParameters=idle_on_node_suspend
 
 # --- Accounting --------------------------------------------------------------
 AccountingStorageType=accounting_storage/slurmdbd

docs/generations.md

@@ -20,8 +20,8 @@ computing teams in a single week might encounter:
 - A cloud-native HPC startup running Ubuntu 24.04 with Slurm 24.05
 
 A demo that only works on one configuration is limited. If you show a customer running
-Slurm 22.05 a Gen 3 demo with `cloud_reg_addrs`, they cannot replicate it — the feature
-does not exist in their version. If you show a Rocky-based demo to an Ubuntu shop, they'll
+Slurm 22.05 a Gen 3 demo that leans on a Slurm 24.05 behavior or cgroup v2-only config,
+they cannot replicate it — those don't exist in their version. If you show a Rocky-based demo to an Ubuntu shop, they'll
 spend the entire meeting translating package names and paths instead of focusing on the
 bursting architecture.
 
@@ -149,17 +149,18 @@ released in May 2025 and major research computing centers are piloting it for ne
 
 **What changes from Gen 2:**
 
-*`cloud_reg_addrs` — the key architectural improvement.* This is the most important
-change from a bursting perspective. In Gen 1 and Gen 2, burst nodes have a specific
-IP address assigned in `slurm.conf` via `NodeAddr=`. When a node boots with a different
-IP (because EC2 assigned a different address from the subnet), slurmctld rejects its
-registration.
+*Burst node addressing — handled by the plugin, not a Slurm parameter.* Burst nodes
+have no `NodeAddr` in `slurm.conf` (and `aws-burst-N` doesn't resolve in DNS). The AWS
+plugin's `resume.py` fills that gap: when a job needs a burst node, it launches the EC2
+instance, reads the assigned private IP, and runs
+`scontrol update nodename=aws-burst-N nodeaddr=<ip> nodehostname=<h>` to tell the
+controller the address — the documented default cloud-registration path (see `cloud_dns`
+in `slurm.conf.5`). This is the same across Gen 1–6 and every Slurm version.
 
-`SlurmctldParameters=cloud_reg_addrs`, added in Slurm 24.05, changes this: when a burst
-node registers, slurmctld updates its `NodeAddr` with the actual IP the node is connecting
-from. No pre-configured `NodeAddr` is needed. The `generate_conf.py` output does not
-include `NodeAddr` lines for burst nodes. This eliminates an entire class of registration
-failures that Gen 1/2 customers encounter.
+> **Note:** earlier versions of this guide called out a `SlurmctldParameters=cloud_reg_addrs`
+> option as a Gen 3 improvement. That was incorrect — it's not a documented Slurm option,
+> it was a no-op on 24.05, and Slurm 25.11 rejects it as defunct. BurstLab no longer sets
+> it on any generation; `resume.py` does the address assignment.
 
 *cgroup v2 only — no v1.* Rocky 10 (RHEL 10) compiles the kernel without
 `CONFIG_CGROUP_V1=y`. The legacy cgroup v1 hierarchy does not exist. Only the unified
@@ -240,8 +241,9 @@ the current state-of-the-art for Ubuntu-based HPC clusters.
 
 **What changes from Gen 3 (Rocky 10 + Slurm 24.05):**
 
-*Same Slurm version, different OS family.* Gen 5 uses the same Slurm 24.05 as Gen 3,
-including `cloud_reg_addrs` for dynamic burst node IP registration. The OS-level
+*Same Slurm version, different OS family.* Gen 5 uses the same Slurm 24.05 as Gen 3.
+Burst node IP registration is handled by the AWS plugin's `resume.py` (which runs
+`scontrol update nodeaddr=...` after launch), same as Gen 3. The OS-level
 differences mirror Gen 4: apt package manager, AppArmor, ufw firewall, ubuntu SSH user.
 
 *Python 3.12.* Ubuntu 24.04 ships Python 3.12 as the default python3 (same version as
@@ -255,7 +257,7 @@ cgroup/v2 plugin compiled with dbus-devel and kernel-headers for full systemd in
 **Relevant customer profiles:**
 - Running Ubuntu 24.04 (Noble Numbat) on head node or compute nodes
 - Slurm 24.05 compiled from source or installed via Ubuntu repos
-- Early adopters of `cloud_reg_addrs` on Ubuntu
+- Teams standardizing on Ubuntu 24.04 + Slurm 24.05
 - Cloud-native HPC teams standardizing on Ubuntu 24.04 LTS
 
 **Files:**
@@ -282,7 +284,7 @@ validating the obol budget-enforcement seam on a live ParallelCluster running Sl
 **What changes from Gen 3 (Rocky 10 + Slurm 24.05):**
 
 *Same feature recipe, newer Slurm.* Gen 6 carries the Gen 3/5 configuration unchanged —
-`cloud_reg_addrs`, cgroup v2 only, `task/cgroup` — onto Slurm 25.11. Slurm 25.11 built
+cgroup v2 only, `task/cgroup` — onto Slurm 25.11. Slurm 25.11 built
 cleanly from source with the Gen 2/3 configure line during obol validation, so the build
 recipe is unchanged; the Slurm version is two minor releases past Gen 3/5's 24.05.
 
@@ -318,10 +320,10 @@ doesn't need it; the overlay concern is tracked in obol#137, not here.
 | CentOS 8 or Rocky 8, Slurm 22.x | **Gen 1** | Exact match — same OS, same Slurm version, same Python workaround |
 | Rocky 8, Slurm 23.x (upgraded) | **Gen 1 or Gen 2** | OS matches Gen 1; Slurm version closer to Gen 2. Start with Gen 1 and note the Slurm version differences. |
 | Rocky 9, AlmaLinux 9, RHEL 9, Slurm 23.x | **Gen 2** | Exact match |
-| Rocky 9, Slurm 24.x (early adopter) | **Gen 2 or Gen 3** | OS matches Gen 2; Slurm version matches Gen 3. Cloud_reg_addrs may not be available in their 24.05 build — verify. |
+| Rocky 9, Slurm 24.x (early adopter) | **Gen 2 or Gen 3** | OS matches Gen 2; Slurm version matches Gen 3. Confirm their cgroup mode and Slurm 24.05 behaviors match the Gen 3 recipe — verify. |
 | Rocky 10, RHEL 10, Slurm 24.05+ | **Gen 3** | Exact match |
 | Ubuntu 22.04, Slurm 23.x | **Gen 4** | Exact match — Ubuntu 22.04 + Slurm 23.11 |
-| Ubuntu 24.04, Slurm 24.05+ | **Gen 5** | Exact match — Ubuntu 24.04 + Slurm 24.05 with cloud_reg_addrs |
+| Ubuntu 24.04, Slurm 24.05+ | **Gen 5** | Exact match — Ubuntu 24.04 + Slurm 24.05, cgroup v2 |
 | Ubuntu (any version), Slurm 22.x or 24.x | **Gen 4 or Gen 5** | Match Slurm version first (Gen 4 for 22.x/23.x, Gen 5 for 24.x), note OS version differences |
 | Not sure / first contact | **Gen 1** | Covers the largest installed base. You can always show Gen 3 diffs once you know their environment. |
 
@@ -365,7 +367,7 @@ Gen 2 (Rocky 9, Slurm 23.11)
 Gen 3 (Rocky 10, Slurm 24.05)
   ├── Python 3.12 → boto3 installs directly
   ├── cgroup v2 only (v1 kernel support removed in RHEL 10)
-  ├── cloud_reg_addrs → burst nodes self-register with actual EC2 IP (KEY IMPROVEMENT)
+  ├── burst node IP set by resume.py (scontrol update nodeaddr) — plugin, not slurm.conf
   ├── iptables-nft (iptables-services removed from RHEL 10)
   └── Ed25519 SSH key (avoids RSA-2048 restriction in RHEL 10 DEFAULT crypto policy)
 ```
@@ -387,7 +389,7 @@ Gen 4 (Ubuntu 22.04 LTS, Slurm 23.11)
 Gen 5 (Ubuntu 24.04 LTS, Slurm 24.05)
   ├── Python 3.12 → boto3 installs directly
   ├── cgroup v2 only
-  ├── cloud_reg_addrs → burst nodes self-register with actual EC2 IP (KEY IMPROVEMENT)
+  ├── burst node IP set by resume.py (scontrol update nodeaddr) — plugin, not slurm.conf
   ├── apt package manager, AppArmor, ufw firewall
   ├── SSH user: ubuntu
   └── FSx Lustre: requires burstlab-lustre (Lustre 2.17.53)
@@ -406,4 +408,4 @@ the directive level:
 - [slurm-gen2-deep-dive.md](slurm-gen2-deep-dive.md) — Gen 2 deltas from Gen 1:
   Rocky 9 OS differences, Slurm 23.11 new directives.
 - [slurm-gen3-deep-dive.md](slurm-gen3-deep-dive.md) — Gen 3 deltas from Gen 2:
-  `cloud_reg_addrs`, cgroup v2 only, Rocky 10 system-level changes.
+  burst-node addressing via resume.py, cgroup v2 only, Rocky 10 system-level changes.

docs/sa-guide.md

@@ -37,9 +37,9 @@ scenario. Match the customer's environment:
 |---|---|---|
 | CentOS 8, Rocky 8, RHEL 8 — Slurm 22.x | **Gen 1** | Exact match — same OS, same Slurm, Python 3.6 boto3 shim |
 | Rocky 9, AlmaLinux 9, RHEL 9 — Slurm 23.x | **Gen 2** | Exact match — Python 3.9 native, cgroup v2 |
-| Rocky 10, RHEL 10 — Slurm 24.05+ | **Gen 3** | Exact match — `cloud_reg_addrs`, cgroup v2 only |
+| Rocky 10, RHEL 10 — Slurm 24.05+ | **Gen 3** | Exact match — cgroup v2 only, iptables-nft |
 | Ubuntu 22.04 — Slurm 23.x | **Gen 4** | Ubuntu match — apt/AppArmor, Python 3.10; EFS validated, FSx client supported (Scenario 4 not yet validated) |
-| Ubuntu 24.04 — Slurm 24.05+ | **Gen 5** | Ubuntu match — apt/AppArmor, Python 3.12, `cloud_reg_addrs` |
+| Ubuntu 24.04 — Slurm 24.05+ | **Gen 5** | Ubuntu match — apt/AppArmor, Python 3.12, cgroup v2 |
 | Not sure / first contact (RHEL) | **Gen 1** | Covers largest RHEL installed base |
 | Not sure / first contact (Ubuntu) | **Gen 4** | Covers largest Ubuntu installed base |
 | CentOS 7 or older RHEL | **Gen 1** as reference | Note OS differences; Slurm/Plugin config identical |