Out of Memory at Test Time

[12dmodel]

Hi,

I'm getting out of memory at test time (16GB on my graphics card). This is true even when I reduce patch size to only 20x20 (x3 channel). However, I was able to train the network at 80x80 and batch size of 32 using only 4.5GB memory.

Do you have any idea what might cause this? I'll look at your code closely in the meantime, but any pointer is greatly appreciated.

I don't have TC installed as I am not using conda. Does TC help you save memory?

Thank you.

[12dmodel]

I was stupid enough not to actually change the input size. However, now I am able to run at 200x200 input size (x 1 batch) at 11GB. It seems like memory consumption doesn't grow linearly with the input size (32x3x80x80 input during training uses only 4.5GB). Is this because there will be more candidate to do NN search, which grows as image_size^2?

[tobiasploetz]

Hi @12dmodel,

I just updated the code with a much more memory efficient implementation of some operations that caused memory bottlenecks so far.

Bests,
Tobias