Regression on read speed

[vthorey]

I was performing some testing on read time with the new version of Highdicom and found it's much slower. Here is the summary of my experiment.

import os
import timeit

import highdicom
import pydicom
from highdicom.io import ImageFileReader

file_path = "./99e90a87-e900-4e76-8805-eb0f785fba14.dcm"

print(f"highdicom.__version__: {highdicom.__version__}")
print(f"pydicom.__version__: {pydicom.__version__}")

ds = pydicom.dcmread(file_path)
original_file_size_mb = round(os.path.getsize(file_path) / 1024 / 1024, 2)
print(f"original file size: {original_file_size_mb}MB nb_frames: {ds.NumberOfFrames}")

def get_dicom_data_iter(fpath):
    dcm = ImageFileReader(fpath)
    dcm.open()
    for i in range(dcm.number_of_frames):
        frame = dcm.read_frame(i, correct_color=hasattr(dcm.metadata, "ICCProfile"))
        yield frame


times = timeit.repeat(lambda: [frame for frame in get_dicom_data_iter(file_path)], repeat=3, number=1)
print(f"avg read time: {sum(times) / len(times):.6f}s")

And the reports

original file size: 136.58MB nb_frames: 714
highdicom.__version__: 0.22.0
pydicom.__version__: 2.4.4
no ICC Profile found in image metadata.
no ICC Profile found in image metadata.
no ICC Profile found in image metadata.
avg read time: 12.218269s

original file size: 136.58MB nb_frames: 714
highdicom.__version__: 0.25.1
pydicom.__version__: 3.0.1
no ICC Profile found in image metadata.
no ICC Profile found in image metadata.
no ICC Profile found in image metadata.
avg read time: 29.315099s

So a factor 3 on this large file. Is there anything I might be doing wrong? This was performed on python 3.10.17

[CPBridge]

Hi @vthorey, thanks for bringing this to our attention

I am not aware of any changes in highdicom that would really explain this issue, so we will need to do some digging. Perhaps you could help with a few things:

• Can you provide any sample files? If not, can you give us some information about the files you are trying to read? Most importantly:
  • Modality
  • SOPClassUID
  • TransferSyntaxUID
  • SamplesPerPixel
  • BitsStored
  • BitsAllocated
  • Whether it has a basic offset table (BOT) or extended offset table (or neither). If you don't know, ExtendedOffsetTable is an attribute in the header that you can easily check for. A basic offset table can be checked for by loading in the whle file with pydicom and checking has_bot = (dcm.Pixeldata[4:8] != b'\x00\x00\x00\x00'). There were some small tweaks around this recently.
• I suspect that this could be a speed regression in pydicom, which we use to handle all the pixel decompression tasks. pydicom 3.0 completely refactored the way pixel data is handled. Could you see whether reading the file with dcm = pydicom.dcmread(...) and then accessing the full pixel matrix with dcm.pixel_array is slower in your newer environment than in your older environment.
• I wonder which part of the process has been slowed down. It could either be the creation of the ImageFileReader object, the open() method (I recommend you use the context manager here instead) or the reading of individual frames (or a combination). Would you be able to time each of these steps individually somehow?

[vthorey]

You were right, I opened an issue on pydicom. There is the dicom file attached to this issue: pydicom/pydicom#2226

[scaramallion]

The difference is because v3.0 now converts YBR data to RGB by default and it's not particularly fast with hundreds of frames. The previous behaviour can be restored with:

ds.pixel_array_options(raw=True)
ds.pixel_array

Returning the raw pixel data then using OpenCV or Pillow to perform the colour space conversion would probably be faster.

[CPBridge]

Thanks @scaramallion !

Although this is primarily a pydicom issue, I'll leave this open for now and monitor the linked pydicom issue. I think this highlights that it may be useful to expose more of the options pydicom gives its users at the highdicom level, something I had already been considering

[fedorov]

It is great that the conversion can be disabled and the original data can still be accessible!