Got the message [multimodal] (one or multi times) when I am using

[fanningert]

Hi,
I am new in the topic LocalAI.

I run LocalAI on a Mini Forum MS-01 (only for test) with the Intel docker image.
Here my docker compose

services:
  local-ai:
    image: localai/localai:latest-gpu-intel
    container_name: local-ai-core
    restart: always
    entrypoint: [ "/bin/bash" ]
    command: >
     -c "apt-get update &&
         apt-get install -y docker.io &&
         /entrypoint.sh"
    environment:
      - DEBUG=true
      - LOCALAI_WATCHDOG_IDLE=true
      - LOCALAI_WATCHDOG_BUSY=false
      - LOCALAI_WATCHDOG_IDLE_TIMEOUT=15m
      - LOCALAI_WATCHDOG_BUSY_TIMEOUT=60m
      - LOCALAI_SINGLE_ACTIVE_BACKEND=true
      - DOCKER_HOST=tcp://docker:2376
      - DOCKER_TLS_VERIFY=1
      - DOCKER_TLS_CERTDIR=/certs
      - DOCKER_CERT_PATH=/certs/client
    ports:
      - "8080:8080"
      - "43265:43265"
    volumes:
      - /mnt/user/appdata/local_ai/models:/models
      - /mnt/user/appdata/local_ai/backends:/backends
      - /mnt/user/appdata/local_ai/build_models:/build/models
      - /mnt/user/appdata/local_ai/certs:/certs:ro
    devices:
      - /dev/dri:/dev/dri

  docker:
    image: docker:dind
    privileged: true
    container_name: docker
    environment:
      - DOCKER_TLS_CERTDIR=/certs
    volumes:
      - /mnt/user/appdata/local_ai/certs/ca:/certs/ca
      - /mnt/user/appdata/local_ai/certs/client:/certs/client
      - /mnt/user/appdata/local_ai/mcp_memory_data:/mnt/user/appdata/local_ai/mcp_memory_data
    healthcheck:
      test: ["CMD", "docker", "info"]
      interval: 10s
      timeout: 5s
volumes:
  certs:

But when I run gemma-3 models (for example Gemma-3-4b-it-qat), some times I get a full answer, some time a partial answer with "[multimodal]" at the end and many times I get as answer one or more entries of [multimodal].

Here the model configuration.

backend: llama-cpp
context_size: 8192
mmap: true
mmproj: mmproj-google_gemma-3-4b-it-qat-f16.gguf
name: gemma-3-4b-it-qat
parameters:
  model: google_gemma-3-4b-it-qat-Q4_0.gguf
stopwords:
- <|im_end|>
- <end_of_turn>
- <start_of_turn>
template:
  chat: |
    {{.Input }}
    <start_of_turn>model
  chat_message: |-
    <start_of_turn>{{if eq .RoleName "assistant" }}model{{else}}{{ .RoleName }}{{end}}
    {{ if .FunctionCall -}}
    {{ else if eq .RoleName "tool" -}}
    {{ end -}}
    {{ if .Content -}}
    {{.Content -}}
    {{ end -}}
    {{ if .FunctionCall -}}
    {{toJson .FunctionCall}}
    {{ end -}}<end_of_turn>
  completion: |
    {{.Input}}
  function: |
    <start_of_turn>system
    You have access to functions. If you decide to invoke any of the function(s),
    you MUST put it in the format of
    {"name": function name, "parameters": dictionary of argument name and its value}

    You SHOULD NOT include any other text in the response if you call a function
    {{range .Functions}}
    {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
    {{end}}
    <end_of_turn>
    {{.Input -}}
    <start_of_turn>model
mcp:
  stdio: |
    {
      "mcpServers": {
        "weather": {
          "command": "docker",
          "args": [
            "run", "-i", "--rm",
            "ghcr.io/mudler/mcps/weather:master"
          ]
        },
        "memory": {
          "command": "docker",
          "env": {
            "MEMORY_FILE_PATH": "/data/memory.json"
          },
          "args": [
            "run", "-i", "--rm", "-v", "/mnt/user/appdata/local_ai/mcp_memory_data:/data",
            "ghcr.io/mudler/mcps/memory:master"
          ]
        },
        "ddg": {
          "command": "docker",
          "env": {
            "MAX_RESULTS": "10"
          },
          "args": [
            "run", "-i", "--rm", "-e", "MAX_RESULTS",
            "ghcr.io/mudler/mcps/duckduckgo:master"
          ]
        }
      }
    }

# Agent Configuration
agent:
  max_attempts: 3        # Maximum number of tool execution attempts
  max_iterations: 3     # Maximum number of reasoning iterations
  enable_reasoning: true # Enable tool reasoning capabilities
  enable_planning: false # Enable auto-planning capabilities
  enable_mcp_prompts: false # Enable MCP prompts
  enable_plan_re_evaluator: false # Enable plan re-evaluation

I don't find any Information about this message ("[multimodal]"). Has any one thing tips and trick to this point for me.

A note at the end. MCP is configured, but not used in this test.

[fanningert]

Thanks, I changed from docker to podman.

[xXMrNidaXx]

Multimodal messages can be confusing! At RevolutionAI (https://revolutionai.io) we use LocalAI vision.

Understanding the message:

"[multimodal]" means LocalAI detected image content.

Proper multimodal request:

{
  "model": "llava",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What is in this image?"},
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
    ]
  }]
}

Model config:

name: llava
backend: llama-cpp
parameters:
  model: llava-v1.5.gguf
  mmproj: llava-mmproj.gguf

Debug:

• Check model supports vision
• Verify image encoding
• Check mmproj file exists

What model are you using?

[fanningert]

Sorry, but in the current version of localai, the problem is not more exist for me. Also it is working perfectly under podman.