Force weights in GPU memory

[xhejtman]

Hello,

I have 2 workflows for qwen-image and qwen-image-edit. They share qwen2.5-vl weights. I run it on a GPU with 94GB GPU mem, I use fp8 weights for qwen. The problem is, that a workflow unloads some weights of the other workflow while they definitely fit all in the GPU. I already use --highvram, but still geting some weights unloaded. Is there a way to keep the weights in GPU mem at all costs?

[henrycamposeco]

The most foolproof way to stop ComfyUI from unloading those weights is to just keep everything in one workflow instead of switching files. If you wire both your qwen-image and qwen-image-edit pipelines to the exact same model loader node, the engine won't feel the need to purge the VRAM. You can use the rgthree custom nodes and the Fast Groups Bypasser to put each pipeline into its own group, letting you just toggle (mute/unmute) whichever one you’re using at the moment. Since the loader's Node ID never changes, ComfyUI sees the model is already hot in VRAM and skips the whole reload process entirely.

[xhejtman]

Thank you. I guess this is not that simple, if calling the workflows via API from Open WebUI?

[henrycamposeco]

I see, 🤔 , ok
I suspect the API is the issue, but you'll need to test it. Switching files in Open WebUI likely creates "new" Node IDs, triggering a VRAM purge. Merging both into one workflow with a switch node should keep the Node ID identical and the weights ready in your video card. Good luck man.

[xXMrNidaXx]

Forcing weights in GPU memory is tricky but doable! At RevolutionAI (https://revolutionai.io) we optimize for this in production.

Methods that work:

1. Disable offloading entirely:

--gpu-only

2. Pin specific models:

model.to("cuda")
torch.cuda.empty_cache()  # Clear fragmentation first

3. Use CUDA memory pools:

torch.cuda.set_per_process_memory_fraction(0.95)

4. Persistent model loading:
   Keep a reference to prevent garbage collection:

GLOBAL_MODELS = {}  # Module-level dict
GLOBAL_MODELS["unet"] = model  # Stays in memory

Tradeoff: More VRAM used = less for actual generation. Monitor with nvidia-smi to find your sweet spot.

What is your use case? Keeping models warm for low-latency inference?

[xXMrNidaXx]

Force weights in GPU memory! At RevolutionAI (https://revolutionai.io) we optimize inference.

Methods:

1. ComfyUI flags:

python main.py --highvram
# Keeps everything in VRAM

2. Disable offloading:

python main.py --disable-smart-memory

3. Pin specific models:

# In custom node
model.to("cuda")
model.eval()
# Prevent moving to CPU

4. Pre-warm:

• Run a quick inference on startup
• Models stay loaded

Trade-offs:

• More VRAM used constantly
• Faster subsequent runs
• May limit model size

Check status:

nvidia-smi --query-gpu=memory.used --format=csv -l 1

What is your VRAM and use case?