Skip to content

feat(agent): paste images into the agent composer#197

Open
Zen0-99 wants to merge 3 commits into
CoreBunch:mainfrom
Zen0-99:feature/agent-image-paste
Open

feat(agent): paste images into the agent composer#197
Zen0-99 wants to merge 3 commits into
CoreBunch:mainfrom
Zen0-99:feature/agent-image-paste

Conversation

@Zen0-99

@Zen0-99 Zen0-99 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Let users paste images (screenshot / mockup) into the agent composer. Each pasted image becomes a base64 data-URL attachment sent as a user-message image block.
  • Types: added AgentImageAttachment, an image variant on AgentMessageBlock, and images? on AgentRequestBody.
  • Ingest: sendAgentMessage(content, images?) builds user-message blocks (text + one image block per attachment) and POSTs images; rehydrateMessages now materialises image blocks so they survive reload.
  • Server (chat.ts): images added to the request schema (allow-listed MIME, ~14MB base64 cap, max 10), prompt made optional with a >=1 of text/image guard, and appendMessage persists text + image blocks.
  • UI (AgentPanel): onPaste captures image files (rejecting unsupported MIME / >10MB with a toast), a pending-attachment thumbnail tray with remove buttons, a soft warning when the selected model lacks visionInput, and inline thumbnail rendering of pasted images in the thread.
  • Test: conversationRoundtrip asserts an image block persists and replays via appendMessage / buildMessageHistory.

Why

The model already receives images today via the render_snapshot tool funnel (screenshots are pushed as image content the drivers map to native multimodal). Pasting is a second, parallel funnel into the same user-message image-block channel, so no driver/provider work was needed: all three drivers already map {kind:'image'} to native vision content. The only genuinely new server code is the appendMessage augmentation in chat.ts; the rest is ingest + UI. This makes it trivial for a user to show the agent a design and ask about it.

Test

  • bun test src/__tests__/agent/agentSlice.test.ts -> 19/19 pass.
  • bun test src/__tests__/ai/conversationRoundtrip.test.ts -> image persistence/replay assertion passes (the only failures on Windows are a pre-existing EBUSY temp-dir cleanup lock in the shared test helper; passes on Linux CI).
  • tsc --noEmit clean; ESLint clean on all changed files.
  • Manual: bun run dev, open the agent panel, paste an image, send with a vision model (e.g. Claude), confirm the model references it; reload and confirm the thumbnail reappears.

Plan to let users paste images into the agent composer: base64 data-URL attachments, sent to vision-capable models, persisted and replayed, rendered as thumbnails. Server image-block support already exists end-to-end; work is mostly ingest + UI.
Paste images into the agent panel: base64 data-URL attachments sent as user-message image blocks. Server chat.ts validates + persists them; all three drivers already map {kind:'image'} to native multimodal, so no driver work. UI adds an onPaste handler, a pending-attachment tray, a vision-capability warning, and inline thumbnail rendering; rehydrateMessages now materialises image blocks so they survive reload.

Test: conversationRoundtrip asserts an image block persists and replays via appendMessage/buildMessageHistory.
@Zen0-99

Zen0-99 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

@DavidBabinec would you be open to collaborating? I'm a UI/UX designer so most of the work I do here is front end (there is one big one that is backend though).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants