Skip to content

Errors during segmentation do not kill subsequent rules #43

Description

@matt-sd-watson

If an error such as mis-specified channel causes the segmentation to fail:

2026-08-20 10:48:07.905799: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2026-08-20 10:48:07.905836: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ltri-ThinkPad-P16-Gen-2): /proc/driver/nvidia/version does not exist
2026-08-20 10:48:07.907286: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.
Error segmenting objects in data/patient1/img/raw/Patient1_001.tiff: Invalid number of aggregated channels: expected 2, got 47
Traceback (most recent call last):
  File "/home/admin/anaconda3/envs/steinbock-snakemake/lib/python3.8/site-packages/steinbock/segmentation/deepcell.py", line 130, in try_segment_objects
    raise SteinbockDeepcellSegmentationException(
steinbock.segmentation.deepcell.SteinbockDeepcellSegmentationException: Invalid number of aggregated channels: expected 2, got 47
Error segmenting objects in data/patient1/img/raw/Patient1_002.tiff: Invalid number of aggregated channels: expected 2, got 47
Traceback (most recent call last):
  File "/home/admin/anaconda3/envs/steinbock-snakemake/lib/python3.8/site-packages/steinbock/segmentation/deepcell.py", line 130, in try_segment_objects
    raise SteinbockDeepcellSegmentationException(
steinbock.segmentation.deepcell.SteinbockDeepcellSegmentationException: Invalid number of aggregated channels: expected 2, got 47
Error segmenting objects in data/patient1/img/raw/Patient1_003.tiff: Invalid number of aggregated channels: expected 2, got 47
Traceback (most recent call last):
  File "/home/admin/anaconda3/envs/steinbock-snakemake/lib/python3.8/site-packages/steinbock/segmentation/deepcell.py", line 130, in try_segment_objects
    raise SteinbockDeepcellSegmentationException(
steinbock.segmentation.deepcell.SteinbockDeepcellSegmentationException: Invalid number of aggregated channels: expected 2, got 47

This does not halt the pipeline from proceeding to the next rules such as quantification or region props on those rules. This can be problematic because if there is a long CLI stack trace of errors, users have difficulty diagnosing at what rules the error occured (it makes it seem like the error occurs at quantification or region props, when in reality it happened d
during segmentation).

I suspect that this is due to the way that the exception is simply logged, but never raised:

https://github.com/BodenmillerGroup/steinbock/blob/main/steinbock/segmentation/deepcell.py#L137

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions