Excessive memory use/leak when training

[fgdfgfthgr-fox]

79 GB of memory is used just for epoch 1. Really? Is this intended or a bug?

minimal_script_dataloaderissues.py

datasplit.csv

Update:

And if you trains for more epochs, it only goes up more.

cellmap-segmentation-challenge repository version: 7a69d9b
cellmap-data version: 2025.7.24.1615

[fgdfgfthgr-fox]

Nevermind, just use the latest version of cellmap-data.

[fgdfgfthgr-fox]

No... Using latest version of cellmap-data helps prevent RAM use from blowing up to inf, but the RAM use is still very high. I am doing mito class, 8 workers, array size of (192, 192, 192) and batch size of 4. It's taking 70GB+ of memory. And I can't reopen the issue.
@rhoadesScholar

[rhoadesScholar]

@fgdfgfthgr-fox Thanks for bringing this to my attention. It sounds like the Xarray-TensorStore backend might not be doing resampling lazily blockwise as intended. I will look into it further as soon as final scoring has been settled.

[fgdfgfthgr-fox]

Final scoring... oh, I thought the challenge deadline has been extended. Guess I am wrong.

[rhoadesScholar]

@fgdfgfthgr-fox I was hoping to extend the deadline, but unfortunately, it is not my decision to make. However, we are planning on continuing to host an evaluation service (with rate limits on uploads) for the community to use to continue work on the challenge. After surveying the results, it turns out this is a hard challenge, and a good benchmark we want to keep around for the community to continue to work towards.

The "official" winners of the challenge period will be announced next week, most likely.