Skip to content

feat(📸): importExternalTexture part 0#372

Merged
wcandillon merged 55 commits into
mainfrom
external-texture-min
Jun 2, 2026
Merged

feat(📸): importExternalTexture part 0#372
wcandillon merged 55 commits into
mainfrom
external-texture-min

Conversation

@wcandillon
Copy link
Copy Markdown
Owner

@wcandillon wcandillon commented Jun 1, 2026

  • Document
  • Implement in Skia

wcandillon and others added 30 commits May 19, 2026 10:03
# Conflicts:
#	apps/example/src/App.tsx
#	apps/example/src/Home.tsx
#	apps/example/src/Route.ts
#	apps/example/src/SharedTextureMemory/SharedTextureMemory.tsx
#	packages/webgpu/android/cpp/AndroidPlatformContext.h
#	packages/webgpu/apple/ApplePlatformContext.h
#	packages/webgpu/apple/ApplePlatformContext.mm
#	packages/webgpu/apple/AppleVideoPlayer.h
#	packages/webgpu/apple/AppleVideoPlayer.mm
#	packages/webgpu/cpp/rnwgpu/PlatformContext.h
#	packages/webgpu/cpp/rnwgpu/api/GPUDevice.h
#	packages/webgpu/cpp/rnwgpu/api/GPUSharedTextureMemory.cpp
#	packages/webgpu/cpp/rnwgpu/api/RNWebGPU.h
#	packages/webgpu/cpp/rnwgpu/api/VideoFrame.h
#	packages/webgpu/cpp/rnwgpu/api/VideoPlayer.h
#	packages/webgpu/src/Canvas.tsx
#	packages/webgpu/src/index.tsx
# Conflicts:
#	apps/example/ios/Podfile.lock
#	apps/example/package.json
#	apps/example/src/App.tsx
#	apps/example/src/Home.tsx
#	apps/example/src/Route.ts
#	yarn.lock
Replace the per-frame fromEquirectangularTexture blit with a real
THREE.CubeCamera that renders a layer-1-only sky sphere wearing the
worklet-updated camera feed. The reflection now picks up any scene
geometry placed on layer 1, not just the panorama itself.

https://claude.ai/code/session_01Sgh5mWoT9XBAYQotQFf94g
The front camera is a narrow-FOV image, not a 360° environment, so wrapping
it equirectangularly produces an obviously fake "your face is the world"
look. Treat the frame as a virtual screen at the viewer's location instead:
a billboarded plane sized at the camera's natural 9:16 aspect that tracks
the orbit camera each frame. The CubeCamera at the helmet's center bakes
that plane into the cube faces, so surfaces facing the viewer pick up the
user's face the way a real chrome object would, while a soft hemisphere-
gradient backdrop fills the rest so grazing reflections don't fall to
black. Env texture changes from 1024×512 (stretched equirect) to 540×960
(matches the rotated frame's aspect — no more distortion).
@wcandillon wcandillon merged commit 9c84ecb into main Jun 2, 2026
2 checks passed
@mrousavy
Copy link
Copy Markdown
Contributor

mrousavy commented Jun 2, 2026

Great stuff!

Just to verify; does the YUV import path support full range on iOS?

I think right now both full range (kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) and video range (kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange) are imported as video range (aka limited color space).

That might produce wrong colors in YUV, especially noticeable in highlights or dark areas. But I might be wrong this is based on an AI summary of the PR

@mrousavy
Copy link
Copy Markdown
Contributor

mrousavy commented Jun 2, 2026

I asked Codex to summarize support for specific pixel formats that VisionCamera can produce:


Checked current main as of June 2, 2026:

  • VisionCamera: bf30b4c
  • react-native-skia: 257e6e6
  • react-native-webgpu: 730010c
  • Relevant open WebGPU PR found: #375, but it adds shared-fence/sync support, not broader pixel-format support.

Legend:

  • Yes: explicit or intended support.
  • Partial: can work only through a narrower path, with caveats.
  • No: not supported correctly by the current NativeBuffer importer.

Android

VisionCamera format Android backing Reachable via VisionCamera Skia NativeBuffer WebGPU NativeBuffer Notes
private ImageFormat.PRIVATE / AHardwareBuffer pixelFormat: "native" Yes Partial Best zero-copy producer path. Skia uses Android external texture import. WebGPU imports zero-copy, but opaque YCbCr currently needs shader-side YUV decode/flip.
yuv-420-8-bit-video YUV_420_888 pixelFormat: "yuv" Partial Partial VisionCamera’s yuv path is CameraX CPU image analysis. NativeBuffer only helps if an AHardwareBuffer exists. WebGPU only recognizes 420 AHB as NV12 if format is Y8Cb8Cr8_420.
yuv-422-8-bit-video YUV_422_888 device format metadata Partial No Skia may import an AHB external texture if one exists. WebGPU wrapper does not classify this as YUV.
yuv-444-8-bit-video YUV_444_888 device format metadata Partial No Same as 422.
yuv-420-10-bit-video YCBCR_P010 device format metadata Partial Partial WebGPU recognizes AHARDWAREBUFFER_FORMAT_YCbCr_P010 as NV12-like, but color/range handling is not complete.
yuv-422-10-bit-video YCBCR_P210 device format metadata Partial No WebGPU wrapper does not classify P210 as YUV.
rgb-rgb-8-bit FLEX_RGB_888 device format metadata Partial Partial Not VisionCamera’s efficient path. Depends on whether an importable AHB exists.
rgb-rgba-8-bit FLEX_RGBA_8888 / CameraX RGBA pixelFormat: "rgb" Partial Partial VisionCamera rgb is CPU conversion. If an RGBA AHB exists, both can import single-plane color.
depth-16-bit DEPTH16 depth output No No Not supported as a drawable NativeBuffer image/external texture.
depth-point-cloud-32-bit DEPTH_POINT_CLOUD depth output No No Point cloud, not image texture interop.
unknown fallback fallback No No No safe import semantics.

iOS

VisionCamera format CoreVideo backing Reachable via VisionCamera Skia NativeBuffer WebGPU NativeBuffer Notes
yuv-420-8-bit-video 420YpCbCr8BiPlanarVideoRange native format Yes Yes WebGPU explicitly recognizes this as NV12.
yuv-420-8-bit-full 420YpCbCr8BiPlanarFullRange pixelFormat: "yuv" / native Yes Partial WebGPU recognizes it as NV12, but current matrix path treats full/video range too loosely.
yuv-420-10-bit-video 420YpCbCr10BiPlanarVideoRange native format Yes No Skia handles 10-bit YUV planes. WebGPU falls back to BGRA classification.
yuv-420-10-bit-full 420YpCbCr10BiPlanarFullRange native format Yes No Same.
yuv-422-8-bit-video 422YpCbCr8BiPlanarVideoRange native format Yes No Skia handles 422. WebGPU only recognizes 420 8-bit NV12.
yuv-422-8-bit-full 422YpCbCr8BiPlanarFullRange native format Yes No Same.
yuv-422-10-bit-video 422YpCbCr10BiPlanarVideoRange native format Yes No Same.
yuv-422-10-bit-full 422YpCbCr10BiPlanarFullRange native format Yes No Same.
yuv-444-8-bit-video 444YpCbCr8BiPlanarVideoRange native format Yes No Skia handles 444. WebGPU does not.
rgb-bgra-8-bit 32BGRA pixelFormat: "rgb" / native Yes Yes This is the safest RGB path for both.
rgb-rgba-8-bit 32RGBA native format Yes Partial Skia explicitly handles RGBA. WebGPU may import as single-plane color, but does not distinguish it in its public NativeVideoPixelFormat.
rgb-rgb-8-bit 24RGB native format No No Skia base classifier sees RGB, but the RGB import path only handles 32BGRA/32RGBA.
raw-bayer-packed96-12-bit 96VersatileBayerPacked12 native/photo RAW No No Not a drawable image texture without demosaic/raw processing.
raw-bayer-unpacked-16-bit 16VersatileBayer native/photo RAW No No Same.
depth-16-bit DepthFloat16 depth output No No Depth object path, not camera texture import.
depth-32-bit DepthFloat32 depth output No No Same.
disparity-16-bit DisparityFloat16 depth output No No Same.
disparity-32-bit DisparityFloat32 depth output No No Same.
unknown fallback fallback No No No safe import semantics.

@mrousavy
Copy link
Copy Markdown
Contributor

mrousavy commented Jun 2, 2026

Realistically speaking, you would only ever use PRIVATE on Android (zero-copy GPU friendly), and 420YpCbCr8BiPlanarVideoRange or 420YpCbCr8BiPlanarFullRange on iOS - and react-native-webgpu supports both of those, so I think this is great and enough already :)

@mrousavy
Copy link
Copy Markdown
Contributor

mrousavy commented Jun 2, 2026

Also I think we can import frames in formats like depth-16-bit into react-native-webgpu using ArrayBuffer APIs, which involves GPU -> CPU -> GPU down/uploads and then describing the texture with WebGPU format/layout props - so I think this either way all works!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants