removed data_list from repository, which was never used for years

Author: Federico Errica

CHANGELOG.md

@@ -1,14 +1,21 @@
 # Changelog
 
-## [1.5.0] Pytorch Automatic Mixed-Precision Support
+## [1.5.0] Pytorch Distributed Data Parallel and Automatic Mixed-Precision Support
 
 ## Added
 
+- added Distributed Data Parallel (DDP) training support that is automatically enabled when `device: cuda` and `gpus_per_task` is an integer greater than 1
+- added DDP-aware data loading through `DistributedSampler` for training splits, so each rank processes a distinct shard
+- added a DDP example configuration file at `examples/MODEL_CONFIGS/config_MLP_ddp.yml` and a lightweight smoke test script at `examples/toy_ddp_training_smoke.py`
 - added optional AMP configuration (`engine.args.mixed_precision`, `engine.args.mixed_precision_dtype`) with dotted dtype paths (e.g. `torch.float16`) and autocast support on CUDA/CPU
 - added configurable ordered model-selection criteria via `model_selection_criteria` (lexicographic comparison with per-criterion direction)
 
+
 ## Changed
 
+- evaluator/experiment execution now forwards progress and termination signals correctly in both debug and non-debug modes when DDP is active
+- distributed helper utilities are now centralized in `mlwiz.training.distributed` for reuse across engine/experiment code paths
+- rank-0-only side effects are enforced for distributed runs (logging/progress/checkpoint-related writes), reducing file contention across ranks
 - model selection now supports metrics from both loss and score aggregates, and non-main metrics must explicitly specify `source: loss|score`
 - configuration validation now raises an error when both `model_selection_criteria` and `higher_results_are_better` are provided
 - updated `MODEL_CONFIGS` examples/integration configs and templates to use `model_selection_criteria` as the default specification

docs/intro.rst

@@ -9,6 +9,7 @@ MLWiz helps you to:
  * reduce the amount of boilerplate code to write,
  * make it flexible enough to encompass a wide range of use cases for research.
  * support a number of different hardware set ups, including a cluster of nodes (using `Ray <https://docs.ray.io/en/latest/>`_),
+ * run single-GPU or multi-GPU training (DDP) transparently through configuration only, with or without mixed-precision
 
 To run an experiment, you usually rely on 2 **YAML configuration files**:
   * one to pre-process the dataset and create the data splits,

docs/tutorial.rst

@@ -114,7 +114,7 @@ explanation of each field as a comment:
     device:  # cpu | cuda
     max_cpus:  # > 1 for parallelism
     max_gpus: # > 0 for gpu usage (device must be cuda though)
-    gpus_per_task:  # percentage of gpus to allocate for each task
+    gpus_per_task:  # Ray GPUs per task: fraction (<=1) or integer (>1 enables DDP)
     gpus_subset: # optional, comma-separated list of gpu indices, e.g. 0,2. Used to force a particular subset of GPUs being used.
 
 
@@ -133,16 +133,16 @@ explanation of each field as a comment:
 
 
     # Experiment
-    result_folder:  # path of the folder where to store results
-    exp_name:  # name of the experiment
-    experiment:  # dotted path to experiment class
-    model_selection_criteria:  # ordered model-selection criteria
-      - metric: main_score
-        direction: max
-    evaluate_every:  # evaluate on train/val/test every `n` epochs and log results
-    risk_assessment_training_runs:  # how many final (model assessment) training runs to perform to mitigate bad initializations
-    model_selection_training_runs:  # how many training runs to perform for each hyper-parameter configuration in a specific inner fold
-    training_timeout_seconds:  # optional max time (in seconds) for a single training run (-1 disables the timeout)
+    result_folder:  # path of the folder where to store results
+    exp_name:  # name of the experiment
+    experiment:  # dotted path to experiment class
+    model_selection_criteria:  # ordered model-selection criteria
+      - metric: main_score
+        direction: max
+    evaluate_every:  # evaluate on train/val/test every `n` epochs and log results
+    risk_assessment_training_runs:  # how many final (model assessment) training runs to perform to mitigate bad initializations
+    model_selection_training_runs:  # how many training runs to perform for each hyper-parameter configuration in a specific inner fold
+    training_timeout_seconds:  # optional max time (in seconds) for a single training run (-1 disables the timeout)
 
     # Grid Search
     # if only 1 configuration is selected, any inner model selection will be skipped
@@ -192,12 +192,12 @@ explanation of each field as a comment:
             main_scorer: mlwiz.training.callback.metric.MulticlassAccuracy
             my_second_metric: mlwiz.training.callback.metric.ToyMetric
 
-      # Training engine
-      engine:
-        class_name: mlwiz.training.engine.TrainingEngine
-        args:
-          mixed_precision: False  # set to True to enable torch AMP autocast (CUDA/CPU)
-          mixed_precision_dtype: torch.float16  # torch.float16 | torch.bfloat16
+      # Training engine
+      engine:
+        class_name: mlwiz.training.engine.TrainingEngine
+        args:
+          mixed_precision: False  # set to True to enable torch AMP autocast (CUDA/CPU)
+          mixed_precision_dtype: torch.float16  # torch.float16 | torch.bfloat16
 
       # Gradient clipper (optional)
       gradient_clipper: null
@@ -244,6 +244,22 @@ Here we can define how many resources to allocate to parallelize different exper
     max_gpus: 2
     gpus_per_task:  0.5
 
+For **Distributed Data Parallel (DDP)**, set ``gpus_per_task`` to an integer greater than 1.
+When ``device: cuda``, MLWiz automatically switches to DDP inside each Ray task.
+
+.. code-block:: yaml
+
+    # one experiment uses 2 GPUs with DDP
+    # with max_gpus: 4, Ray can run up to 2 such experiments in parallel
+    device: cuda
+    max_cpus: 24
+    max_gpus: 4
+    gpus_per_task: 2
+
+MLWiz shards the training data with ``DistributedSampler`` and keeps a single set of
+experiment artifacts (rank 0 writes logs/checkpoints/plots). If a rank fails, check
+``ddp_rank_0.log``, ``ddp_rank_1.log``, ... inside the run folder.
+
 
 
 Data Loading
@@ -271,26 +287,26 @@ our results:
 
 .. code-block:: yaml
 
-    result_folder: RESULTS
-    exp_name: mlp
-    experiment: mlwiz.experiment.MLP
-    model_selection_criteria:
-      - metric: main_score
-        direction: max
-    evaluate_every: 3
-    risk_assessment_training_runs: 3
-    model_selection_training_runs: 2
-    training_timeout_seconds: -1  # set to a positive value to enforce a per-run time budget
-
-``higher_results_are_better`` is still supported as a legacy shortcut for a
-single ``main_score`` criterion, but it cannot be used together with
-``model_selection_criteria``.
-
-By default MLWiz will run each training session until either the configured number of epochs is reached or the early
-stopper halts it. If you need to cap the wall-clock time of each run, set ``training_timeout_seconds`` to a positive
-value. The :class:`~mlwiz.training.engine.TrainingEngine` tracks the elapsed time (including previous attempts when
-resuming from checkpoints) and stops scheduling additional epochs once the limit is reached, logging the reason for the
-interruption. Keeping checkpointing enabled lets you safely resume from where the timeout was triggered.
+    result_folder: RESULTS
+    exp_name: mlp
+    experiment: mlwiz.experiment.MLP
+    model_selection_criteria:
+      - metric: main_score
+        direction: max
+    evaluate_every: 3
+    risk_assessment_training_runs: 3
+    model_selection_training_runs: 2
+    training_timeout_seconds: -1  # set to a positive value to enforce a per-run time budget
+
+``higher_results_are_better`` is still supported as a legacy shortcut for a
+single ``main_score`` criterion, but it cannot be used together with
+``model_selection_criteria``.
+
+By default MLWiz will run each training session until either the configured number of epochs is reached or the early
+stopper halts it. If you need to cap the wall-clock time of each run, set ``training_timeout_seconds`` to a positive
+value. The :class:`~mlwiz.training.engine.TrainingEngine` tracks the elapsed time (including previous attempts when
+resuming from checkpoints) and stops scheduling additional epochs once the limit is reached, logging the reason for the
+interruption. Keeping checkpointing enabled lets you safely resume from where the timeout was triggered.
 
 
 Grid Search
@@ -301,6 +317,7 @@ you can define lists associated to an hyper-parameter and all possible combinati
 nesting of these combinations for maximum flexibility.
 
 There is one config file ``examples/MODEL_CONFIGS/config_MLP.yml`` that you can check to get a better idea.
+For a multi-GPU DDP setup, refer to ``examples/MODEL_CONFIGS/config_MLP_ddp.yml``.
 
 
 Random Search
@@ -332,7 +349,8 @@ or
 
 .. code-block:: bash
 
-    mlwiz-exp --config-file examples/MODEL_CONFIGS/config_MLP.yml
+    mlwiz-exp --config-file examples/MODEL_CONFIGS/config_MLP_ddp.yml
+
 
 
 And we are up and running!
@@ -357,35 +375,35 @@ to you, please consider reading `Samy Bengio's lecture (Part 3) <https://bengio.
 Navigating the live progress UI
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The progress screen is interactive. Press ``:`` to open the small prompt in the bottom-right corner, type a command, and
-hit ``Enter`` to switch what is rendered without stopping the run. Useful commands:
-
-- ``:`` (or ``:g`` / ``:global``): go back to the default overview with all progress bars.
-- ``:r`` (or ``:refresh``): redraw the currently selected view (handy if the terminal layout gets messy).
-- ``:<outer> <run>`` (e.g., ``:1 2``): focus the *risk assessment* run number ``run`` of outer fold ``outer`` (numbers start at 1).
-- ``:<outer> <inner> <config> <run>`` (e.g., ``:2 1 3 1``): focus a *model selection* run for a specific config inside an outer/inner fold pair.
-
-Global overview (default view):
-
-.. image:: _static/exp_gui.png
-   :width: 600
-
-If an identifier is invalid or the run has not produced updates yet, MLWiz will print a short hint and keep listening so
-you can try again.
-
-You can also use the arrow keys:
-
-- **left/right**: move across runs/configurations within the currently selected view.
-- **up/down**: toggle between the most recently visited model selection view and risk assessment view.
-
-Focused run view (same as what you see when running with ``--debug``):
-
-.. image:: _static/run_view.png
-   :width: 600
-
-To stop the computation, use ``CTRL-C`` to send a ``SIGINT`` signal, and consider using the command ``ray stop`` to stop
-all Ray processes. **Warning:** ``ray stop`` stops **all** ray processes you have launched, including those of other
-experiments in progress, if any.
+The progress screen is interactive. Press ``:`` to open the small prompt in the bottom-right corner, type a command, and
+hit ``Enter`` to switch what is rendered without stopping the run. Useful commands:
+
+- ``:`` (or ``:g`` / ``:global``): go back to the default overview with all progress bars.
+- ``:r`` (or ``:refresh``): redraw the currently selected view (handy if the terminal layout gets messy).
+- ``:<outer> <run>`` (e.g., ``:1 2``): focus the *risk assessment* run number ``run`` of outer fold ``outer`` (numbers start at 1).
+- ``:<outer> <inner> <config> <run>`` (e.g., ``:2 1 3 1``): focus a *model selection* run for a specific config inside an outer/inner fold pair.
+
+Global overview (default view):
+
+.. image:: _static/exp_gui.png
+   :width: 600
+
+If an identifier is invalid or the run has not produced updates yet, MLWiz will print a short hint and keep listening so
+you can try again.
+
+You can also use the arrow keys:
+
+- **left/right**: move across runs/configurations within the currently selected view.
+- **up/down**: toggle between the most recently visited model selection view and risk assessment view.
+
+Focused run view (same as what you see when running with ``--debug``):
+
+.. image:: _static/run_view.png
+   :width: 600
+
+To stop the computation, use ``CTRL-C`` to send a ``SIGINT`` signal, and consider using the command ``ray stop`` to stop
+all Ray processes. **Warning:** ``ray stop`` stops **all** ray processes you have launched, including those of other
+experiments in progress, if any.
 
 Useful Features to Know About
 ------------------------------

mlwiz/experiment/experiment.py

@@ -696,13 +696,10 @@ def _run_valid_impl(
             (
                 train_loss,
                 train_score,
-                _,  # check the ordering is correct
                 val_loss,
                 val_score,
                 _,
                 _,
-                _,
-                _,
             ) = training_engine.train(
                 train_loader=train_loader,
                 validation_loader=val_loader,
@@ -824,13 +821,10 @@ def _run_test_impl(
             (
                 train_loss,
                 train_score,
-                _,
                 val_loss,
                 val_score,
-                _,
                 test_loss,
                 test_score,
-                _,
             ) = training_engine.train(
                 train_loader=train_loader,
                 validation_loader=val_loader,

mlwiz/static.py

@@ -76,7 +76,6 @@
 
 # STRING FORMATTING
 TENSORBOARD = "tensorboard"
-EMB_TUPLE_SUBSTR = "_embeddings_tuple"
 ATOMIC_SAVE_EXTENSION = ".part"
 CLASS_NAME = "class_name"
 ARGS = "args"