How to combine model and mmproj ?

[SwHaraday]

Hi,

I've been playing with Qwen3VL-8B-Instruct-Q8_0.gguf on Win11+conda+cuda environment.
It seems to work just for text question but images are not given to model.

Anyone please educate me how to combine mmproj-GGUF to model ?
Just showing link to related information will be appreciated.
Code to load model is as below.

from llama_cpp import Llama
llm = Llama(
    model_path="./Qwen3VL-8B-Instruct-Q8_0.gguf",
    mmproj_path="./mmproj-Qwen3VL-8B-Instruct-Q8_0.gguf",
    n_ctx=1000,
    n_gpu_layer=-1,
    verbose=True, 
)

environment:
Windows11 pro 24H2
GPU NVIDIA RTX3090
python 3.11
torch 2.5.1+cu124
llama_cpp_python 0.3.23

Thank you in advance.

[Floating-Pin]

Hi, I'm using llama-server and a PowerShell script.
It's not exactly what you were looking for, but here is how I run gemma-3-27b with mmproj on Windows:

./main/cmake_native_gpu_release/llama-server `
--model ./models/gemma-3-27b-it-abliterated.q8_0.gguf `
--mmproj ./models/mmproj-mlabonne_gemma-3-27b-it-abliterated-f16.gguf `
--host 0.0.0.0 `
--port 8082 `
--threads 6 `
--flash-attn 1 `
--no-mmap `
--direct-io `
--ctx-size 16384 `
--batch-size 512 `
--ubatch-size 256 `
--jinja `
--n-gpu-layers 7 `
--n-predict -1 `
--keep -1 `
--temp 1 `
--top-k 64 `
--top-p 0.95

[SwHaraday]

Hi Floating-Pin,

Thank you for your reply.
I've checked that your command line works with llama.cpp server.
Besides, my plan is to use Qwen3-VL GGUF model with python script.

There is 'mmproj' option in loading model, but there is no info regarding how and when to use this option on 'llama-cpp-python'.

I know model is working with GPU beccause if I ask 'Please discribe about this image. If image is not provided, let us know', then model answers 'No image is provided.'

Should I ask this somewhere else ?

[Floating-Pin]

You might want to ping the maintainer of the Python wrapper or ask on r/LocalLLaMA.
Alternatively, you could run the llama server and interact with it through the API in your script.

[95zhaohanqing]

已收到

[SwHaraday]

Hi there,
I found what I exactly want to know at "https://github.com/JamePeng/llama-cpp-python?tab=readme-ov-file" ,
section 'Loading a Local Image With Qwen3VL(Thinking/Instruct)'.

Now code works fine.
Hope this can be a foothold for those who walking same muddy path.

[rohitpharande]

Thanks for this info. I was wondering about the same. But did you find any way to merge mmproj and model weights into a single file?

[95zhaohanqing]

已收到