Is 25G VRAM a normal usage ?

[Tyrannas]

Hello,

Trying you model on a GPU with 16Go VRAM, thought it would be ok since it's only a 3B model but I am getting the folowwing error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.55 GiB. GPU 0 has a total capacity of 14.74 GiB of which 6.20 GiB is free.

I only tried the demo you provided:

git clone https://github.com/stanford-oval/churro.git
cd churro
curl -fsSL https://pixi.sh/install.sh | bash
pixi shell -e minimal
python churro_transformers_infer.py my_image.png --max-new-tokens 40

also tried reducing the max new tokens but that does not seem to have any effect.

So I was wondering: is this a normal VRAM usage for the model ?

Thank you

[Tyrannas]

Apparently resizing my image used way less vram (was 4000*3000)

[s-jse]

Thanks for trying Churro!

Yes, this is unfortunately expected for a full, uncompressed, unquantized 3B model. Even though it is "only" 3B parameters, memory use during inference can still be quite high, so running out of VRAM on a 16 GB GPU is normal.

There are third-party quantized versions available, for example this GGUF release: https://huggingface.co/mradermacher/churro-3B-GGUF. We have not tested those ourselves, so we cannot guarantee their behavior or quality, but they may be worth trying if you are running into VRAM issues.

As you mentioned, reducing the input image size can also help, but it may come at the cost of OCR quality, especially for small or low-resolution text.

[Tyrannas]

Ok thx I'll try to experiment a bit with the quantized version !

[Tyrannas]

For anyone here interested, it is amazingly faster and lighter. I get the same output in 10s on 16Go VRAM than on 3min on 24GB with the quantized version.

[s-jse]

Thanks for testing those models! Would you mind sharing which quant model(s) worked well for you?

[Tyrannas]

Sure :)

I used Q4_K_M served with llama.cpp, I suspect the use of llama.cpp instead of torch is also a reason for the speedup, have not tried other models since the Q4_K_M is the one recommended by the author and it was pretty fast on my device.