Repository navigation
fix(background-image): load the image on React Native - #6
Merged
Merged
Conversation
… failures React Native's Blob cannot be created from an ArrayBuffer, so decodeImage threw on every image and the effect silently published the camera without a background. Decode from the bytes when a Blob cannot be made, and report image load errors through onStatus.
MiloszFilimowski
added a commit
to fishjam-cloud/documentation
that referenced
this pull request
Oct 1, 2026
## Description
The React Native docs now send users to camera effects on the camera
Fishjam already manages, instead of VisionCamera. VisionCamera stays as
an integration.
An effect is a camera track middleware. Peers keep receiving it as
`peer.cameraTrack`:
```ts
const blur = createCameraEffectMiddleware(createBackgroundBlurEffect(() => ({ segmentation, radius: 24 })));
setCameraTrackMiddleware(isBlurOn ? null : blur);
```
**Menu changes**
- **Tutorials:** new "Add background blur to your camera". "Draw WebGPU
effects into your published camera" is rewritten on
`createCameraFrameProcessorSession`, with no VisionCamera.
- **How-to → Camera effects (new):** overview, background effects (blur
and image), WebGPU effects, frame processing in a worklet, and stream
middleware (moved here, now covering React Native too).
- **How-to → Custom sources:** keeps non-camera content (web, React
Native, low-level frame API).
- **Integrations → Vision Camera (new):** the VisionCamera tutorial,
how-to, and WebGPU-on-VisionCamera guide, moved from the pages above.
- **Concepts:** new "How camera effects work". "How custom sources work"
links to it.
- **Redirects** for the four moved pages.
**Build and API reference**
- `web-client-sdk` submodule bumped to fff4e570 (web-client-sdk#606).
- New submodules `video-effects` (v0.1.5) and `react-native-worklets`
(v0.12.1), with TypeDoc pages.
- `typegpu` pinned to 0.12.4 (the version `video-effects` uses). The
VisionCamera WebGPU snippet maps `typegpu` to the SDK's 0.11 through
hidden Twoslash flags, so every snippet still type-checks.
## Tested
- All snippets type-check, `yarn build` passes with no broken links, and
the pre-commit checks (format, spellcheck, typecheck) pass.
- Ran the tutorial code end to end in a test app: iOS simulator (SimCam)
and Galaxy S23, debug and release builds, staging room. Blur, grayscale
+ watermark and the frame monitor work, and each device receives the
other's effect.
Testing found these, now fixed in the docs:
- Android needs `minSdkVersion: 26` (`react-native-webgpu`). The docs
add it with `expo-build-properties`.
- In Android release builds, `expo-asset` keeps bundled images as
drawable resources, so reading them with `File` fails. The docs include
a workaround, checked in a release build.
- `tgpu.resolve({ externals })` returns empty WGSL in TypeGPU 0.12. The
examples use the array form.
Performance on the Galaxy S23 (release build, 720x1280): blur runs at 30
fps with no dropped frames at about 110% CPU of 800% and 33% GPU; the
WebGPU tutorial effect at about 111% CPU and 12% GPU. Graphics memory
showed a 200 MB to 1 GB saw-tooth with any effect on
`react-native-webgpu` 0.5.15. The cause was in `react-native-webgpu`
(`GPUExternalTexture.destroy()` did not release the imported frame);
fixed upstream in wcandillon/react-native-webgpu#478, released in
0.10.1, memory flat at about 145 MB after it.
Fixes opened in our repos along the way:
- fishjam-cloud/video-effects#6: background images load on React Native
(released in 0.1.4)
- fishjam-cloud/video-effects#7: frames that skip `render()` no longer
hold GPU memory (released in 0.1.4)
- fishjam-cloud/video-effects#9: own WebGPU flag constants, because
`react-native-webgpu` 0.8.4 and newer export them as `undefined`
(released in 0.1.5). The WebGPU tutorial uses the `GPUTextureUsage` /
`GPUShaderStage` globals for the same reason.
- fishjam-cloud/web-client-sdk#610: fishjam-chat sets Android
`minSdkVersion` 26 and moves to `react-native-webgpu` 0.10.1
The docs require `@fishjam-cloud/react-native-client` 0.30.2 or newer,
`@fishjam-cloud/video-effects` 0.1.5 or newer and `react-native-webgpu`
0.10.1 or newer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background image works on React Native
The background image effect never showed an image on React Native. The camera was published unchanged, with no error.
Why:
decodeImagewraps the image bytes in aBlobwheneverBlobexists. React Native has aBlobglobal, but it throws for anArrayBuffer:The error was swallowed in
ensureImage, so the effect stayed a passthrough. This happened for bothimage.uriandimage.data.Fix:
decodeImagefalls back to passing the bytes straight tocreateImageBitmapwhen aBlobcan't be made. Browsers still get aBlob, andreact-native-webgputakes theArrayBuffer.onStatusas"error". It is reported after the segmentation status settles, so"ready"doesn't hide it. The effect still falls back to passthrough, as before.CHANGELOG has an Unreleased entry.
Checked
yarn test(tsc, eslint, prettier) andyarn build.distin a test app:createImageBitmap(new Blob([bytes]))throws the error above, and no background is drawn.image.datareplaces the background.onStatus("error", "Couldn't decode image")after"ready".Found while testing the React Native camera effects docs.