feat(plugin): support images in custom tool responses#9542
feat(plugin): support images in custom tool responses#9542leeweisern wants to merge 1 commit intoanomalyco:devfrom
Conversation
Custom tools can now return images alongside text using a rich content format:
return {
content: [
{ type: "text", text: "Screenshot captured" },
{ type: "image", mimeType: "image/png", data: base64 }
]
}
- Add ToolContentPart and ToolResult types to plugin
- Add content normalization helper with validation
- Handle rich content in tool registry (backwards compatible)
- Add attachments field to tool.execute.after hook
- Add unit tests with 100% coverage on content module
- Update docs with usage examples and limits
Limits: max 10 images, ~5MB per image, PNG/JPEG/GIF/WebP only.
Fixes anomalyco#9539
|
The following comment was made by an LLM, it may be inaccurate: No duplicate PRs found |
|
@rekram1-node Could you help add the |
|
this seems overcomplicated, we already have the attachments notion I think someone else has a similar pr that accomplishes this |
you mean it can support returning images in custom tool? |
|
I think this PR would solve a problem I'm having. I'm having Claude (via opencode) write Playwright tests, but when a test fails, Claude can't see the screenshots in the trace. I've written a command-line tool that supports commands like |
00637c0 to
71e0ba2
Compare
f1ae801 to
08fa7f7
Compare
|
Closing this pull request because it has had no updates for more than 60 days. If you plan to continue working on it, feel free to reopen or open a new PR. |
|
My problem is actually solved - claude can save the image as a file and view it without my intervention. |
Summary
Custom tools can now return images alongside text, allowing the LLM to actually see and analyze visual data.
{ content: [{ type: "text", text }, { type: "image", mimeType, data }] }Changes
ToolContentPartandToolResulttypes to pluginattachmentsfield totool.execute.afterhookVerification
bun test test/tool/registry.test.ts --test-name-pattern "tool.content"content.tsFixes #9539