feat(agent): paste images into the agent composer#197
Open
Zen0-99 wants to merge 3 commits into
Open
Conversation
Plan to let users paste images into the agent composer: base64 data-URL attachments, sent to vision-capable models, persisted and replayed, rendered as thumbnails. Server image-block support already exists end-to-end; work is mostly ingest + UI.
Paste images into the agent panel: base64 data-URL attachments sent as user-message image blocks. Server chat.ts validates + persists them; all three drivers already map {kind:'image'} to native multimodal, so no driver work. UI adds an onPaste handler, a pending-attachment tray, a vision-capability warning, and inline thumbnail rendering; rehydrateMessages now materialises image blocks so they survive reload.
Test: conversationRoundtrip asserts an image block persists and replays via appendMessage/buildMessageHistory.
Contributor
Author
|
@DavidBabinec would you be open to collaborating? I'm a UI/UX designer so most of the work I do here is front end (there is one big one that is backend though). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AgentImageAttachment, animagevariant onAgentMessageBlock, andimages?onAgentRequestBody.sendAgentMessage(content, images?)builds user-message blocks (text + one image block per attachment) and POSTsimages;rehydrateMessagesnow materialises image blocks so they survive reload.chat.ts):imagesadded to the request schema (allow-listed MIME, ~14MB base64 cap, max 10),promptmade optional with a>=1 of text/imageguard, andappendMessagepersists text + image blocks.AgentPanel):onPastecaptures image files (rejecting unsupported MIME / >10MB with a toast), a pending-attachment thumbnail tray with remove buttons, a soft warning when the selected model lacksvisionInput, and inline thumbnail rendering of pasted images in the thread.conversationRoundtripasserts an image block persists and replays viaappendMessage/buildMessageHistory.Why
The model already receives images today via the
render_snapshottool funnel (screenshots are pushed as image content the drivers map to native multimodal). Pasting is a second, parallel funnel into the same user-message image-block channel, so no driver/provider work was needed: all three drivers already map{kind:'image'}to native vision content. The only genuinely new server code is theappendMessageaugmentation inchat.ts; the rest is ingest + UI. This makes it trivial for a user to show the agent a design and ask about it.Test
bun test src/__tests__/agent/agentSlice.test.ts-> 19/19 pass.bun test src/__tests__/ai/conversationRoundtrip.test.ts-> image persistence/replay assertion passes (the only failures on Windows are a pre-existingEBUSYtemp-dir cleanup lock in the shared test helper; passes on Linux CI).tsc --noEmitclean; ESLint clean on all changed files.bun run dev, open the agent panel, paste an image, send with a vision model (e.g. Claude), confirm the model references it; reload and confirm the thumbnail reappears.