Repository navigation
fix(react-native): release camera imports for frames the kernel skips - #7
Merged
Merged
Conversation
A frame kernel may skip render() to drop a frame, but the camera frame was already imported as an external texture. Dawn releases that import only once queue work completes, so with nothing submitted each skipped frame kept its camera buffer alive. Submit an empty command list for skipped frames.
MiloszFilimowski
added a commit
to fishjam-cloud/documentation
that referenced
this pull request
Oct 1, 2026
## Description
The React Native docs now send users to camera effects on the camera
Fishjam already manages, instead of VisionCamera. VisionCamera stays as
an integration.
An effect is a camera track middleware. Peers keep receiving it as
`peer.cameraTrack`:
```ts
const blur = createCameraEffectMiddleware(createBackgroundBlurEffect(() => ({ segmentation, radius: 24 })));
setCameraTrackMiddleware(isBlurOn ? null : blur);
```
**Menu changes**
- **Tutorials:** new "Add background blur to your camera". "Draw WebGPU
effects into your published camera" is rewritten on
`createCameraFrameProcessorSession`, with no VisionCamera.
- **How-to → Camera effects (new):** overview, background effects (blur
and image), WebGPU effects, frame processing in a worklet, and stream
middleware (moved here, now covering React Native too).
- **How-to → Custom sources:** keeps non-camera content (web, React
Native, low-level frame API).
- **Integrations → Vision Camera (new):** the VisionCamera tutorial,
how-to, and WebGPU-on-VisionCamera guide, moved from the pages above.
- **Concepts:** new "How camera effects work". "How custom sources work"
links to it.
- **Redirects** for the four moved pages.
**Build and API reference**
- `web-client-sdk` submodule bumped to fff4e570 (web-client-sdk#606).
- New submodules `video-effects` (v0.1.5) and `react-native-worklets`
(v0.12.1), with TypeDoc pages.
- `typegpu` pinned to 0.12.4 (the version `video-effects` uses). The
VisionCamera WebGPU snippet maps `typegpu` to the SDK's 0.11 through
hidden Twoslash flags, so every snippet still type-checks.
## Tested
- All snippets type-check, `yarn build` passes with no broken links, and
the pre-commit checks (format, spellcheck, typecheck) pass.
- Ran the tutorial code end to end in a test app: iOS simulator (SimCam)
and Galaxy S23, debug and release builds, staging room. Blur, grayscale
+ watermark and the frame monitor work, and each device receives the
other's effect.
Testing found these, now fixed in the docs:
- Android needs `minSdkVersion: 26` (`react-native-webgpu`). The docs
add it with `expo-build-properties`.
- In Android release builds, `expo-asset` keeps bundled images as
drawable resources, so reading them with `File` fails. The docs include
a workaround, checked in a release build.
- `tgpu.resolve({ externals })` returns empty WGSL in TypeGPU 0.12. The
examples use the array form.
Performance on the Galaxy S23 (release build, 720x1280): blur runs at 30
fps with no dropped frames at about 110% CPU of 800% and 33% GPU; the
WebGPU tutorial effect at about 111% CPU and 12% GPU. Graphics memory
showed a 200 MB to 1 GB saw-tooth with any effect on
`react-native-webgpu` 0.5.15. The cause was in `react-native-webgpu`
(`GPUExternalTexture.destroy()` did not release the imported frame);
fixed upstream in wcandillon/react-native-webgpu#478, released in
0.10.1, memory flat at about 145 MB after it.
Fixes opened in our repos along the way:
- fishjam-cloud/video-effects#6: background images load on React Native
(released in 0.1.4)
- fishjam-cloud/video-effects#7: frames that skip `render()` no longer
hold GPU memory (released in 0.1.4)
- fishjam-cloud/video-effects#9: own WebGPU flag constants, because
`react-native-webgpu` 0.8.4 and newer export them as `undefined`
(released in 0.1.5). The WebGPU tutorial uses the `GPUTextureUsage` /
`GPUShaderStage` globals for the same reason.
- fishjam-cloud/web-client-sdk#610: fishjam-chat sets Android
`minSdkVersion` 26 and moves to `react-native-webgpu` 0.10.1
The docs require `@fishjam-cloud/react-native-client` 0.30.2 or newer,
`@fishjam-cloud/video-effects` 0.1.5 or newer and `react-native-webgpu`
0.10.1 or newer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skipping render() no longer grows GPU memory without bound
CameraFrameKernelis documented so that skippingrender(...)drops the frame. On Android, a kernel that skips frames makes graphics memory grow by about one camera frame per frame, and it never comes back.Why:
renderFrameimports the camera as an external texture before it calls the kernel.cameraTexture.destroy()andvideoFrame.release()run, but Dawn frees the imported buffer only after queue work completes. With no submit at all, nothing is ever freed.Fix: when the kernel did not call
render, submit an empty command list, so Dawn processes the pending releases.Checked
yarn test(tsc, eslint, prettier).render, graphics memory fromdumpsys meminfo:Not fixed here
Even kernels that render every frame show a saw-tooth: 300 MB → 1.1 GB during the WebGPU tutorial effect, and about 400 MB still held after the effect is removed. Per-frame GPU objects appear to be freed only on GC or Dawn ticks inside
react-native-webgpu. Setting_instance/_sourceto null inGPUExternalTexture::destroy()did not change it. That needs a look inreact-native-webgpuitself.Found while measuring the React Native camera effects docs examples.