Feat/native call lifecycle spike - #1481
Draft
eleboucher wants to merge 47 commits into
Draft
Conversation
eleboucher
marked this pull request as draft
July 29, 2026 14:12
eleboucher
force-pushed
the
feat/native-call-lifecycle-spike
branch
from
July 29, 2026 15:07
1daf36f to
6992828
Compare
eleboucher
marked this pull request as ready for review
July 30, 2026 11:58
eleboucher
marked this pull request as draft
July 30, 2026 11:58
Fold in semantic follow-ups from rebasing onto origin/dev: - generated tauri bindings: take dev's regeneration wholesale; the spike nets no surviving typegen output (its vendored lifecycle plugin is removed again), and dropping it silently removed dev's start/stopCallForegroundService bindings - restore dev's bytes exactly. - Cargo.lock: reset to dev's resolution (fixes auto-merge downgrades of windows-sys/toml/libloading) and re-add only tauri-plugin-livekit-mobile at 77d263f via cargo metadata. - ios-project.yml: drop the spike's duplicate NSMicrophone/NSCameraUsageDescription; dev already ships its own wording for both keys, and YAML duplicate keys would override dev's text. - Cargo.toml: drop a leftover blank line from conflict resolution.
eleboucher
force-pushed
the
feat/native-call-lifecycle-spike
branch
from
July 30, 2026 12:19
866cda8 to
9ae03b1
Compare
- resolve LiveKit participants to Matrix users for names and avatars - show participants in audio calls instead of an empty grid - add a device preview and apply the chosen devices when publishing - apply prescreen mic/camera/sound choices on both backends - gate publishing on E2EE readiness - show device errors, add speaker selection - derive publish controls from the token grants - stop a failed session blocking calls in other rooms - keep a remote key from masking a local import failure - only fall back to the legacy provisioning endpoint on 404/405 - restore the recovery-key guard dropped by an earlier rebase
Contributor
Deploying with
|
| Status | Preview URL | Commit | Alias | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! | https://pr-1481-sable.raspy-dream-bb1d.workers.dev | f2afeec | pr-1481 |
Fri, 31 Jul 2026 11:29:16 GMT |
tauri-plugin-livekit-mobile links the LiveKit Swift SDK through SPM, and that package already pins webrtc-xcframework 144.7559.11 and livekit-uniffi-xcframework 0.0.6, the same builds the script fetched. Listing them again under bundle.iOS.frameworks embedded a second copy of both. Also declare the audio background mode the plugin requires for call audio to survive backgrounding.
- centre the audio-call layout; the column had no width so it hugged the left - resolve the local participant from the client, and index the legacy user:device identity alongside the anonymised one, so tiles stop reading "Unknown participant" - log an unresolved identity with the keys we do know - give the speaker menu an icon and the device menus a caret; MediaDeviceMenu renders an empty button and leaves the affordance to CSS - stop the muted-red style leaking onto device menu buttons via aria-pressed
CallStatusRenderer only knew about the Element Call embed, so the bottom bar disappeared entirely on the LiveKit path. Reuse the existing bar (live chip, room name, member glance and the control buttons) driven by the LiveKit local participant. Sound moves to a shared atom so the bar and the call surface agree on it.
…, Matrix Call 2.0 fixes
…filmstrip, icon disconnect, quality dot
…lity on participants
… path RTCEncryptionManager emits our own key under `userId:deviceId` unless unstableSendStickyEvents is set, which we never set, while CallMembership.rtcBackendIdentity is a SHA-256 for sticky-event memberships. The native controller was already fixed; the JS path still passed the hashed value, so the provider never matched its own key, e2eeReady stayed false, and the control bar and initial media publish never happened. Also green the suite and lint: point the native controller fixtures at the real own identity, follow the hangup button's rename to "End call", type the lifecycle test mocks, and attach rejection handlers before flushing timers so vitest stops reporting unhandled errors.
The native surface derived a label by string-parsing the LiveKit identity, so tiles showed a raw MXID and initials cut from it, with no avatar. Reuse the identity map and profile lookup the LiveKit JS surface already uses. Also include connectionQuality in the participant equality check; without it a snapshot that only changed quality was dropped and the quality dot went stale.
A remote hangup arrives as an idle snapshot, which was passed to cleanup as a failure, so the user got a red "Call failed" screen reading "Native call ended." with a Dismiss button. Clear the session instead, as the LiveKit JS path already does. Failure snapshots now map NativeCallFailureCode to user-facing copy, and setup errors no longer append the raw exception: a user could be shown a TypeError message. The detail is logged instead.
…x keyboard reveal CallStatusRenderer never consulted nativeCallAtom, so navigating away from the room during a native call left no way to hang up. Extract the bar shell both engines share and add a native control cluster. On mobile the call view defaults to 30% height, sized for the Element Call participant list. A native call renders video tiles and a control bar in that space, which clipped the floating local preview and collapsed grid rows to name strips; give it at least 75%. The web control bar hides itself with visibility:hidden, which removes it from the tab order, and nothing else inside the surface is focusable, so the existing focus reveal could never fire from a Tab press. Reveal on keydown too, and let the control pill wrap so End call cannot be clipped off a narrow viewport. The test that covered this synthesised focusIn on the surface and passed for a scenario the browser cannot produce.
Sliding sync only ships m.room.member for $ME, so isValidMembership discarded every RTC membership including our own, leaving session.memberships empty. The join survived only because waitForOwnMembership polls server state, but the empty roster also left RTCEncryptionManager with no recipients, so media keys were never distributed to the other participants.
livekit-client defaults adaptiveStream and dynacast to false, so a multi-party call received every published layer at full quality and kept sending video nobody displayed. The native engine already enables both. ConnectionFeedback also treated SignalReconnecting and Connecting as a lost connection, showing critical red while media was still flowing.
…etime The room page stays mounted for the whole mobile slide transition, so geometry alone kept the native video painted over the outgoing page. Drive overlay visibility off route selection instead, and give the remote overlay the occlusion check the local one already had. getAudioRoutes and setAudioRoute were implemented natively but never called, so wire them to a control that only appears once the platform reports routes. Feature a remote screen share over a camera, since iOS already projects one. Media toggles now follow the connected state, which is what the native setters require. The quality dot carries state through colour alone, so give it a label and Sable's own tokens.
Add the JitPack repository that livekit-android's audioswitch dependency is published on, enable the core library desugaring the plugin AAR requires, and resolve the protobuf-java/protobuf-javalite duplicate classes in favour of the full runtime that tink's generated code needs. These live in Tauri-generated files, so tauri android init would drop them.
The identifier was switched to moe.sable.dev for local device signing, which orphans src-tauri/gen/android and makes the Android build fail.
The dependency was a path override pointing at a local checkout, which breaks CI and every other machine.
eleboucher
force-pushed
the
feat/native-call-lifecycle-spike
branch
from
July 31, 2026 19:41
f2afeec to
0ef377d
Compare
The plugin renamed its commands to snake_case, which Tauri's docs prescribe, so the invoke strings move with it. sendDTMF is gone from the plugin: it had no caller and only played a CallKit tone.
… access Keeping the camera running in Picture in Picture needs AVCaptureSession's multitasking camera access. Below iOS 16 that requires the com.apple.developer.avfoundation.multitasking-camera-access entitlement, which has no public request process, so 15.0 could not support it.
The native video view sits above the webview, so the route picker opened underneath it and looked like nothing happened. The occlusion probe cannot catch this: it only runs on resize, scroll and intersection, and a menu appearing above the slot moves nothing.
The call canvas, tiles, control pill and menus were painted with hardcoded dark hex values, so the call ignored the active theme. They now use the same tokens as the rest of the app. The scrim over a video tile and its text stay light on dark, because they sit on video rather than on a themed surface, as does the hangup button. The route picker rendered the wire's category marker as the option name, so the list read current and input. It now shows the platform's display name and ticks the active route.
…r the oldest transport The app published legacy memberships while requesting the modern /get_token endpoint first. Those endpoints grant different LiveKit participant identities, and LiveKit binds E2EE keys strictly by identity, so against an SFU service that implements /get_token the call would connect, fill the roster and decrypt nothing in either direction. It worked only because the deployment falls back. One flag now drives both the endpoint and the join config so they cannot drift, and the identity comes from the EncryptionKeyChanged event rather than being reconstructed. focus_active advertises oldest_membership, which obliges everyone to use the oldest member's transport. We always used our own, so a peer whose homeserver advertises a different SFU shared a Matrix session with us and no media at all. Memberships also expired after four hours with no delayed events and no leave on page hide, leaving ghost participants that suppressed the ring for later joiners.
The local tile rendered a generic person icon while remote tiles resolved a Matrix profile, so with the camera off your own tile was the only one without an avatar. The profile hook already resolves the local user from its isLocal argument whatever identity the SFU handed us.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #
Type of change
Checklist:
AI disclosure: