Skip to content

Refactor scan grouping - #1061

Draft
tsalo wants to merge 48 commits into
mainfrom
refactor-scan-grouping
Draft

Refactor scan grouping#1061
tsalo wants to merge 48 commits into
mainfrom
refactor-scan-grouping

Conversation

@tsalo

@tsalo tsalo commented Jun 11, 2026

Copy link
Copy Markdown
Member

Keeping as a draft for now.

Open issues

  • DRBUDDI RPE.

Summary

Reorganizes QSIPrep's scan-grouping and field-map-selection code into SDCFlows-style value objects, without changing behavior. The previously monolithic qsiprep/utils/grouping.py is split into a clear pipeline and shrinks from 1725 → 1055 lines.

  • New qsiprep/utils/fieldmaps.py (+217) mirroring sdcflows.fieldmaps: EstimatorType, MODALITIES, FieldmapFile (sibling discovery), and FieldmapEstimation (method inference, bids_id, and to_fieldmap_info() — the single place per-suffix field-map knowledge now lives).
  • Slimmed grouping.py: DwiSeries → _final_distortion_groups → find_estimators (B0FieldIdentifier → IntendedFor →
    PE-heuristic) → _serialize_estimation_groups + build_fmap_application_groups, with group_dwi_scans reduced to a thin orchestrator.
  • base.py adapter builds fieldmap_info via FieldmapEstimation.to_fieldmap_info() instead of re-deriving suffix typing.
  • Deleted the post-hoc refine/remap dance (refine_distortion_groups, remap*) and the duplicated per-suffix helpers (classify_fmap_files, get_highest_priority_fieldmap, FMAP_PRIORITY, the legacy build_distortion_groups/build_fmap_estimation_groups and their helpers) — ~670 lines net removed.

Why it's safe (behavior preservation)

  • Adds a golden/characterization suite (test_grouping_golden.py, 240 snapshots) frozen from the pre-refactor
    group-dwi-scans baseline, covering every fixture × combine_scans/ignore_fieldmaps/estimate_per_axis combo, including error cases. The new code reproduces it exactly and deterministically.
  • For the trickiest edge cases (same-physical-signature multi-run DWIs with per-run field-map links via
    B0FieldSource/IntendedFor), the four-dict output was verified byte-identical to the pre-refactor baseline across all 8
    flag combos, locked by two dedicated fixtures.
  • Also fixed pre-existing nondeterminism in three grouping error messages (Python set reprs reordered between runs), so the suite is stable.

Test plan

  • pytest qsiprep/tests/test_grouping_golden.py qsiprep/tests/test_utils_grouping.py qsiprep/tests/test_utils_fieldmaps.py — all green.
  • Full non-integration suite: 310 passed, 8 xfailed (the 4 test_interfaces_{dipy,freesurfer,mrtrix3} errors are
    pre-existing/unrelated).
  • CircleCI integration jobs (forrest_gump, maternal_brain_project, drbuddi_*) — the end-to-end check that the
    preserved fieldmap_info drives the same workflows.

Notes

  • One test is marked xfail: test_drbuddi_rpe_pair_collapses_to_single_output asserts an unimplemented DRBUDDI behavior (collapsing an opposite-PE DWI into its partner as an rpe_series field map). It fails identically on the pre-refactor baseline — not a regression, but a real feature gap worth a separate issue.
  • fieldmaps.py intentionally mirrors sdcflows so a future offload of distortion correction becomes "add the dependency, delete our copy, delegate" rather than a rewrite.

tsalo and others added 30 commits August 7, 2025 09:07
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tsalo and others added 18 commits June 10, 2026 15:23
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A phase2 file is always a sibling of phase1, never an estimation primary.
Removing it from _GRE_SUFFIXES prevents a degenerate phase2-only source set
from inferring PHASEDIFF and then raising StopIteration in to_fieldmap_info.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Locks the cross-session split behavior of named estimation groups
(pepolar01 spanning two sessions -> pepolar01_ses-01 / pepolar01_ses-02)
which no existing fixture exercised.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reproduce the legacy _split_named_member_groups_by_session behavior: a named
(B0FieldIdentifier / IntendedFor) estimation group that spans multiple sessions
is split into per-session estimators (e.g. pepolar01 -> pepolar01_ses-01 /
pepolar01_ses-02), and application is derived independently from B0FieldSource.
Locked by the dset_b0field_multisession golden fixture and a find_estimators
unit test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…auto_id order

Review fixes for find_estimators:
- Sort the PE-axis and concatenation-group sets in ValueError messages so the
  golden snapshots are deterministic (set repr ordering was flaky run-to-run).
- _intendedfor_buckets returns buckets in first-appearance (sorted-fmap) order,
  matching the legacy auto_NNNNN numbering instead of sorted-target order.
- Drop a source-less IntendedFor session partition instead of raising
  'Insufficient sources' (degenerate cross-session intent).
- Add dset_intendedfor_autoid_order fixture locking the auto_id ordering.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire find_estimators into group_dwi_scans via the new ordering:
- _final_distortion_groups folds the estimator id into the physical signature,
  so a distortion group never spans two estimators (replaces build-then-refine).
- _serialize_estimation_groups produces the legacy fmap_estimation_groups dict;
  application is derived by build_fmap_application_groups (preserving the exact
  many-to-many B0FieldSource mapping a one-to-one map could not represent).
- IntendedFor estimators include their target series so the estimation group
  records the target dg ids, matching legacy _build_intendedfor_groups.
- FieldmapEstimation gains an explicit bids_id param so a source carrying
  several B0FieldIdentifiers (belonging to several groups) is not flagged as a
  conflict; the axis check reads PE directly instead of building an estimation.
- _as_list unpacks tuples (DwiSeries normalizes list identifiers to tuples) and
  distortion_signature normalizes a scalar B0FieldIdentifier to a 1-tuple,
  matching the legacy tuple(_ensure_list(...)) key.

All 224 golden snapshots stay green and deterministic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code review found a behavior divergence: legacy collapses DWI runs sharing a
physical signature into one distortion group BEFORE resolving field-map links,
so per-run fmaps merge; the series-level estimator-fold split them instead.

- _final_distortion_groups groups by physical signature alone (legacy
  build_distortion_groups). The legacy refine step is a no-op in practice
  (same-signature series share B0FieldIdentifier and resolve IntendedFor to the
  same group), so the estimator fold was wrong and is removed.
- Distortion groups are now built before find_estimators, which takes them and
  resolves IntendedFor against whole distortion groups (dg-level bucketing),
  collapsing per-run fmaps into one estimation group exactly as legacy did.
- find_estimators drops the vestigial series_to_estimator return; application is
  handled wholly by build_fmap_application_groups.
- Add dset_same_sig_b0source and dset_same_sig_intendedfor fixtures + snapshots,
  verified byte-identical to legacy across all 8 flag combos each.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
No tests referenced the deleted internal helpers (all use the public API), so
the only port needed was the pre-existing drbuddi failure: opposite-PE DWI runs
form separate distortion groups and are not collapsed via an rpe_series field
map. This fails identically on the pre-refactor baseline; marked xfail to
document the desired behavior without falsely asserting it works.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Delete three now-unreachable functions ruff cannot detect:
- _get_fmap_files (superseded by _subject_fmap_files)
- _get_distortion_group_multipart_key (its only caller was deleted)
- _group_by_sessions (pre-existing orphan; sole user of get_bids_params)
and drop the now-unused get_bids_params import. Verified by repo-wide grep
(no source or test references) and the full grouping/golden/fieldmaps suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant