Feature: Auto-route pasted images to multimodal-looker when main model lacks vision #5666
itisvincent
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Problem
When a user pastes an image into an opencode session running a non-vision model (e.g.
glm-5.2,deepseek-v4, etc.), opencode sends the image directly to the main model API which rejects it with an error like:This happens before any OMO agent logic runs, so the
multimodal-lookeragent (which exists exactly for this purpose) never gets a chance to handle the image.Current behavior
ollama-cloud/glm-5.2)The
look_attool exists and routes tomultimodal-lookerautomatically, but only when explicitly invoked. There is no hook that intercepts pasted images and routes them.Proposed solution
Add a hook (e.g.
chat.message.beforeorsession.error) that detects:modalities.inputincludesimage)When both conditions are met, auto-invoke the
look_attool with the pasted image data, routing it to themultimodal-lookeragent — which already has a vision-capable model configured (e.g.ollama-cloud/gemma4:31b).Alternative: intercept at
session.errorWhen the main model returns an error containing "does not support image input" or similar, automatically retry by routing the image through
look_atinstead.Alternative: pre-check model capabilities
Before sending the message, check if the main model supports image input. If not, strip images from the message and invoke
look_atseparately, then combine the result.Why this matters
glm-5.2) as their main model still need vision occasionallylook_at, or switch the entire session to a vision modelmultimodal-lookeragent,look_attool, vision-capable model cache) — it just needs the auto-routing hookEnvironment
ollama-cloud/glm-5.2as main modelmultimodal-lookerconfigured withollama-cloud/gemma4:31bHappy to help test a prototype if this gets picked up.
All reactions