uto3DSeg SegResNet: Dice peaks 0.18 at epoch ~180 then decays to ~0.08 by epoch 1250 (sparse multi-label pelvic MRI)

[andrubrown]

Hello!

I'm segmenting 5 venous structures (IVC + 4 iliac veins) in pelvic bTFE MRI and hitting a climb then collapse pattern I can't resolve, and I'd appreciate a read on whether this is an LR-schedule / small-data signature or something else.

Auto3DSeg result (SegResNet, single fold):
Validation Dice climbs early, peaks at best_avg_dice 0.182 at epoch 179, then decays over the remaining epochs to ~0.077 by epoch 1250. Training loss keeps converging throughout. So the model demonstrably learns some signal early and then loses it.

Data:

• 39 training cases, single-channel bTFE MRI, 5 foreground labels + background.
• Anisotropic, 6.0 × 0.69 × 0.69 mm spacing.
• Structures are small and sparse. I already cropped each case to an ROI around the labels, raising foreground from ~0.08% to ~1.4% of the volume; the result above is on that cropped data.
• Labels verified non-empty and voxel-aligned to images; Auto3DSeg's own data analysis ran clean (it only flagged non-uniform spacing).

What I've already ruled out:

• Hardware/precision: a standalone 3D U-Net (plain PyTorch) trains a synthetic sparse-blob task to Dice 0.999 in fp32, fp16-AMP, and bf16-AMP on this exact machine; conv3d matches CPU to 1e-5. So the RTX 5090 / CUDA 12.8 / torch 2.11 stack trains segmentation correctly.
• Framework-specific behavior: nnU-Net v2 (2d, 3d_fullres default, 3d_fullres ResEnc, and a high foreground-oversampling variant) shows the same pattern on the same data. One run climbed to EMA pseudo-Dice ~0.06 then collapsed to ~0. So this reproduces across two independent self-configuring pipelines.
• Sparsity as sole cause: the 17× foreground increase from cropping did not change the outcome.

Environment: MONAI 1.6.0, torch 2.11.0+cu128, CUDA 12.8, RTX 5090 (sm_120), Python 3.13, Ubuntu 24.04.

Questions:

1. Is peak-then-decay (best at epoch ~180 of 1250) a recognized LR-schedule or overfitting signature on small, sparse-label datasets and is the fix a shorter schedule / early stopping on the validation metric, a lower LR, or a different loss (Tversky/Focal-Tversky)?
2. For 5 sparse structures in 39 cases, would you recommend region-based training, a two-stage localize-then-segment cascade, or heavier augmentation over tuning the single-stage run?
3. Anything Auto3DSeg-specific I should override (num_epochs, LR, loss) for a dataset this small rather than relying on its auto-configured defaults?

Happy to share the datastats.yaml, hyper_parameters.yaml, and the full training log.

Thanks for your help!