Skip to content

Commit 57f522f

Browse files
juliethecaoELin2025claudeCopilotAnish Shivamurthy
authored
feat(frontend): add HuggingFace media output rendering in result panel (#5675)
### What changes were proposed in this PR? Render image, audio, and video outputs from HuggingFace tasks inline in the workflow result panel instead of displaying raw data URLs as text. **New file — `media-type.util.ts`:** - `isImageUrl` — detects `data:image/` data URLs and common image extensions (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`) - `isAudioUrl` — detects `data:audio/` data URLs and common audio extensions (`.mp3`, `.wav`, `.ogg`, `.m4a`, `.flac`) - `isVideoUrl` — detects `data:video/` data URLs, common video extensions (`.mp4`, `.webm`, `.ogg`), and the `fal.media` CDN host used by fal.ai text-to-video outputs **Changes to `result-table-frame.component.{ts,html}`:** - Add `isImageCell` / `isAudioCell` / `isVideoCell` methods that delegate to the detection helpers - Show a media type icon (play/image/audio) in the table cell; the actual `<img>`, `<audio controls>`, `<video controls>` elements render inline in the row detail modal - Precompute each cell's media type once per row when result data arrives (`cellMediaTypes[rowIndex][columnIndex]`), instead of re-running the detection helpers and getCell on every change-detection cycle **Changes to `result-panel-modal.component.{ts,html}`:** - Build `rowEntries` with per-field media metadata on modal open - Render media fields inline in the row detail view with a copy-to-clipboard fallback for the raw URL - Fetch remote media through the backend's /huggingface/media-proxy endpoint (with JWT auth) rather than loading cross-origin URLs directly in the browser - On a proxy failure (e.g. the backend's SSRF allowlist rejects the URL), fall back to the plain-text view instead of loading the raw remote URL directly - Revoke allocated blob URLs on `ngOnDestroy` to avoid leaking object URLs for the life of the page **Changes to `result-panel-model.component.scss`:** - Add modal-toolbar and row-detail styles for media display **Changes to `operator-property-edit-frame.component.ts`:⋆ - Show validation messages (`validation.show = true`) for the image/audio upload and prompt-column fields on HuggingFace operators **New file — `media-type.util.spec.ts`:** - 22 unit tests covering all three helpers across data URL prefixes, file extensions (including case-insensitivity and query strings), the `fal.media` CDN URL and empty/plain-string inputs ### Any related issues, documentation, discussions? - Tracking issue: #5674 - Closes: #5674 - Stacked on: #5568 - Parent issue: #5041 ### How was this PR tested? - `media-type.util.spec.ts` (22 cases): data URL prefixes, file extensions, case-insensitivity, query strings, the fal.media CDN URL, cross-type rejection, and non-string input handling - `result-table-frame.component.spec.ts`: media cell icon rendering — Play Video/Play Audio/View Image indicators and plain-text fallback - `result-panel-modal.component.spec.ts`: row-entry construction from data URLs vs. remote URLs, blob-fetch success/failure, the SSRF-safe proxy-failure fallback, clipboard copy, and inline video/image `src` binding - Run with `ng test` ### Was this PR authored or co-authored using generative AI tooling? Co-authored with Claude Sonnet 4.6 in compliance with ASF guidelines --------- Signed-off-by: Elliot Lin <36275109+ELin2025@users.noreply.github.com> Signed-off-by: Julie Cao <116243642+juliethecao@users.noreply.github.com> Co-authored-by: Elliot <36275109+Falcons-Royale@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Elliot Lin <36275109+ELin2025@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Anish Shivamurthy <anish@uci.edu> Co-authored-by: Anish Shivamurthy <anishshiva7@gmail.com> Co-authored-by: PG1204 <vidya0dhar@gmail.com> Co-authored-by: Matthew B. <mgball@uci.edu>
1 parent d8d87a9 commit 57f522f

15 files changed

Lines changed: 1340 additions & 46 deletions
Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
/**
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
import { isAudioUrl, isImageUrl, isVideoUrl } from "./media-type.util";
21+
22+
describe("isImageUrl", () => {
23+
it("should return true for data:image/ data URLs", () => {
24+
expect(isImageUrl("data:image/png;base64,abc123")).toBe(true);
25+
expect(isImageUrl("data:image/jpeg;base64,abc123")).toBe(true);
26+
expect(isImageUrl("data:image/webp;base64,abc123")).toBe(true);
27+
});
28+
29+
it("should return true for common image file extensions", () => {
30+
expect(isImageUrl("https://example.com/photo.png")).toBe(true);
31+
expect(isImageUrl("https://example.com/photo.jpg")).toBe(true);
32+
expect(isImageUrl("https://example.com/photo.jpeg")).toBe(true);
33+
expect(isImageUrl("https://example.com/photo.gif")).toBe(true);
34+
expect(isImageUrl("https://example.com/photo.webp")).toBe(true);
35+
});
36+
37+
it("should be case-insensitive for extensions", () => {
38+
expect(isImageUrl("https://example.com/photo.PNG")).toBe(true);
39+
expect(isImageUrl("https://example.com/photo.JPG")).toBe(true);
40+
});
41+
42+
it("should return true for URLs with query strings", () => {
43+
expect(isImageUrl("https://example.com/photo.png?v=1")).toBe(true);
44+
});
45+
46+
it("should return false for audio and video URLs", () => {
47+
expect(isImageUrl("data:audio/mp3;base64,abc")).toBe(false);
48+
expect(isImageUrl("data:video/mp4;base64,abc")).toBe(false);
49+
expect(isImageUrl("https://example.com/clip.mp4")).toBe(false);
50+
});
51+
52+
it("should return false for plain text strings", () => {
53+
expect(isImageUrl("hello world")).toBe(false);
54+
expect(isImageUrl("")).toBe(false);
55+
});
56+
});
57+
58+
describe("isAudioUrl", () => {
59+
it("should return true for data:audio/ data URLs", () => {
60+
expect(isAudioUrl("data:audio/mp3;base64,abc123")).toBe(true);
61+
expect(isAudioUrl("data:audio/wav;base64,abc123")).toBe(true);
62+
});
63+
64+
it("should return true for common audio file extensions", () => {
65+
expect(isAudioUrl("https://example.com/clip.mp3")).toBe(true);
66+
expect(isAudioUrl("https://example.com/clip.wav")).toBe(true);
67+
expect(isAudioUrl("https://example.com/clip.ogg")).toBe(true);
68+
expect(isAudioUrl("https://example.com/clip.m4a")).toBe(true);
69+
expect(isAudioUrl("https://example.com/clip.flac")).toBe(true);
70+
});
71+
72+
it("should be case-insensitive for extensions", () => {
73+
expect(isAudioUrl("https://example.com/clip.MP3")).toBe(true);
74+
expect(isAudioUrl("https://example.com/clip.WAV")).toBe(true);
75+
});
76+
77+
it("should return true for URLs with query strings", () => {
78+
expect(isAudioUrl("https://example.com/clip.mp3?token=xyz")).toBe(true);
79+
});
80+
81+
it("should return false for image and video URLs", () => {
82+
expect(isAudioUrl("data:image/png;base64,abc")).toBe(false);
83+
expect(isAudioUrl("data:video/mp4;base64,abc")).toBe(false);
84+
expect(isAudioUrl("https://example.com/photo.png")).toBe(false);
85+
});
86+
87+
it("should return false for plain text strings", () => {
88+
expect(isAudioUrl("hello world")).toBe(false);
89+
expect(isAudioUrl("")).toBe(false);
90+
});
91+
});
92+
93+
describe("isVideoUrl", () => {
94+
it("should return true for data:video/ data URLs", () => {
95+
expect(isVideoUrl("data:video/mp4;base64,abc123")).toBe(true);
96+
expect(isVideoUrl("data:video/webm;base64,abc123")).toBe(true);
97+
});
98+
99+
it("should return true for common video file extensions", () => {
100+
expect(isVideoUrl("https://example.com/clip.mp4")).toBe(true);
101+
expect(isVideoUrl("https://example.com/clip.webm")).toBe(true);
102+
expect(isVideoUrl("https://example.com/clip.ogv")).toBe(true);
103+
});
104+
105+
it("should return true for fal.media CDN URLs", () => {
106+
expect(isVideoUrl("https://v3b.fal.media/files/abc123/output.mp4")).toBe(true);
107+
});
108+
109+
it("should be case-insensitive for extensions", () => {
110+
expect(isVideoUrl("https://example.com/clip.MP4")).toBe(true);
111+
expect(isVideoUrl("https://example.com/clip.WEBM")).toBe(true);
112+
});
113+
114+
it("should return true for URLs with query strings", () => {
115+
expect(isVideoUrl("https://example.com/clip.mp4?t=5")).toBe(true);
116+
});
117+
118+
it("should return false for image and audio URLs", () => {
119+
expect(isVideoUrl("data:image/png;base64,abc")).toBe(false);
120+
expect(isVideoUrl("data:audio/mp3;base64,abc")).toBe(false);
121+
expect(isVideoUrl("https://example.com/photo.jpg")).toBe(false);
122+
});
123+
124+
it("should return false for plain text strings", () => {
125+
expect(isVideoUrl("hello world")).toBe(false);
126+
expect(isVideoUrl("")).toBe(false);
127+
});
128+
129+
it("should return false for non-string types", () => {
130+
expect(isVideoUrl(null as unknown as string)).toBe(false);
131+
expect(isVideoUrl(undefined as unknown as string)).toBe(false);
132+
expect(isVideoUrl(42 as unknown as string)).toBe(false);
133+
});
134+
});
135+
136+
describe("non-string type guard (shared)", () => {
137+
it("isAudioUrl should return false for non-string types", () => {
138+
expect(isAudioUrl(null as unknown as string)).toBe(false);
139+
expect(isAudioUrl(undefined as unknown as string)).toBe(false);
140+
expect(isAudioUrl(true as unknown as string)).toBe(false);
141+
});
142+
143+
it("isImageUrl should return false for non-string types", () => {
144+
expect(isImageUrl(null as unknown as string)).toBe(false);
145+
expect(isImageUrl(undefined as unknown as string)).toBe(false);
146+
expect(isImageUrl([] as unknown as string)).toBe(false);
147+
});
148+
});
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
/**
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
export function isVideoUrl(value: string): boolean {
21+
if (typeof value !== "string") return false;
22+
return (
23+
/^https?:\/\/.+\.(mp4|webm|ogv)(\?.*)?$/i.test(value) ||
24+
value.startsWith("data:video/") ||
25+
/^https:\/\/[a-z0-9-]+\.fal\.media\/files\//i.test(value)
26+
);
27+
}
28+
29+
export function isAudioUrl(value: string): boolean {
30+
if (typeof value !== "string") return false;
31+
return /^https?:\/\/.+\.(mp3|wav|ogg|m4a|flac)(\?.*)?$/i.test(value) || value.startsWith("data:audio/");
32+
}
33+
34+
export function isImageUrl(value: string): boolean {
35+
if (typeof value !== "string") return false;
36+
return /^https?:\/\/.+\.(png|jpg|jpeg|gif|webp)(\?.*)?$/i.test(value) || value.startsWith("data:image/");
37+
}

0 commit comments

Comments
 (0)