feat(pipeline): submit parallel steps in waves of array jobs

[mpvanderschelling]

Summary

When a parallel step's open-job count exceeds max_array_size * max_jobs_per_task, the orchestrator now submits it as multiple sequential waves of array jobs instead of one strided array. The declared SlurmResources therefore always cover exactly max_jobs_per_task experiments (default 1) per array task, independent of how many open jobs the step has — previously, time/mem silently had to cover ceil(n_open / max_array_size) sequential experiments, a number not even knowable when the config is written.

Design rationale, rejected alternatives (consolidate-and-recount between waves, afterok wave chaining, opt-in default), and the frozen-snapshot invariant are recorded in docs/adr/0002-wave-based-array-submission.md; the Wave term is defined in CONTEXT.md.

Behavior

• SlurmResources.max_jobs_per_task: int | None = 1 — new field, validated in __post_init__. None opts out and restores the single strided submission, rendering byte-identical scripts to the previous renderer (guarded by golden files captured from develop before this change).
• Wave assignment reads the frozen central-store snapshot; task k of wave b owns open[b*W*j:][k::W][:j] — the direct generalization of the previous stride expression (b=0, j=None reproduces it exactly). Per-sample result JSONs are deliberately not consulted between waves: race-free, deterministic wave count, no mid-step mutation of the central store.
• Waves chain internally with afterany (an infra-killed task must not orphan the remaining waves); the next pipeline step's dependency attaches to the final wave only.
• The orchestrator gains a fourth positional counter WAVE_COUNT; the wave index reaches workers via exported F3DASM_WAVE (the F3DASM_ITERATION mechanism) → run_step --wave → ExecutionContext.wave. All wave plumbing is rendered only when some step actually uses waves.
• Behavior is unchanged whenever n_open <= max_array_size (the common case): each task already evaluated exactly one job there.

Tests

• Bash execution harness (test_orchestrator_exec.py): runs rendered orchestrators under bash with a fake sbatch + stubbed count_open, feeding each sbatch "$SELF" … resubmission back in as the next wake. Asserts the full submission trace: per-wave array widths (incl. ragged last wave), afterany chaining, final-wave afterok advance, F3DASM_WAVE values, wave-counter resets across loop iterations. Skipped on Windows.
• Golden byte-identity (test_golden_scripts.py + golden/): max_jobs_per_task=None reproduces the pre-wave renderer byte-for-byte.
• Partition property tests on ExecutionContext.assigned_jobs (waves × tasks partition the open jobs: disjoint, complete, each ≤ j), --wave CLI threading, SlurmResources validation, and string assertions for the new render paths.

pytest (1125 passed, coverage 91.8%), ruff check, pre-commit run --all-files, and mkdocs build all pass locally.

Release

Version bumped to 2.3.0 (new public field + deliberate behavioral change in the overflow regime). Rollback for users who want the old strided behavior: max_jobs_per_task=None.

🤖 Generated with Claude Code