From 2efdc7650336526c698aa5410865e79e49d3057c Mon Sep 17 00:00:00 2001 From: lucas Date: Sat, 17 Oct 2026 11:36:37 +0800 Subject: [PATCH 1/2] fix(portal): bootstrap video for iPhone locked auto unlock --- AGENTS.md | 1 + README.en.md | 8 +- README.md | 8 +- docs/experiments/issue-29-locked-video.md | 98 ++++++ src/locked_video.rs | 101 +++++++ src/locked_video_tests.rs | 350 ++++++++++++++++++++++ src/main.rs | 1 + src/video.rs | 4 +- src/webrtc.rs | 5 +- 9 files changed, 567 insertions(+), 9 deletions(-) create mode 100644 docs/experiments/issue-29-locked-video.md create mode 100644 src/locked_video.rs create mode 100644 src/locked_video_tests.rs diff --git a/AGENTS.md b/AGENTS.md index 68cc726..a12ccb9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -128,6 +128,7 @@ every 60 s. | `src/auth.rs`, `src/config.rs`, `src/login.rs`, `src/login_ui.rs` | signed REST client, config, device init + QR login, GTK window | | `src/webrtc.rs` | peer connection, data channels, CONTROL dispatch, initialization messages | | `src/video.rs`, `src/capture_setting.rs`, `src/video_track.rs`, `src/hevc.rs` | FFmpeg H.264/H.265 pipeline, one video track for both codecs (RFC 7798 HEVC payloader); viewer quality/fps/resolution/chroma settings | +| `src/locked_video.rs` | low-rate synthetic black video on the negotiated track while Portal is pending and locked; enables iPhone cold auto unlock (#29), requires FFmpeg libx264/libx265 for the selected codec | | `src/audio.rs` | Pulse/PipeWire monitor → Opus audio track | | `src/input.rs`, `src/input_mapping.rs`, `src/uinput.rs` | input injection (X11 XTest / Wayland / portal / uinput), Windows VK → keysym → evdev | | `src/cursor.rs` | pointer shape mirroring (X11) | diff --git a/README.en.md b/README.en.md index 8a8e9a8..bef0226 100644 --- a/README.en.md +++ b/README.en.md @@ -67,7 +67,7 @@ authorized session. Independent file/terminal tool peers do not reset input. | Hyprland | Wayland (native wlroots protocols) | 🧪 Experimental, tested in a VM (some features limited, see below) | `--features wayland` | | GNOME / KDE Plasma | Wayland (xdg-desktop-portal) | 🧪 Experimental; validation scope and limits below | `--features wayland,portal` | -The GNOME Wayland Portal/EIS path has been checked for video and mouse/keyboard input on Ubuntu 26.04, GNOME 50, with one 4K screen at 200% scale; auto unlock after a cold locked connection is verified with the official Mac viewer, while other lock recovery and reconnection still need hardware acceptance testing. Apart from that path and GNOME on X11, desktop testing has primarily used virtual machines. If something goes wrong on +The GNOME Wayland Portal/EIS path has been checked for video and mouse/keyboard input on Ubuntu 26.04, GNOME 50, with one 4K screen at 200% scale; auto unlock after a cold locked connection is verified with the official Mac and iPhone 4.42.0 viewers, while other lock recovery and reconnection still need hardware acceptance testing. Apart from that path and GNOME on X11, desktop testing has primarily used virtual machines. If something goes wrong on real hardware, please [open an issue](../../issues) with your distribution, desktop, GPU and the relevant lines from `journalctl --user -u uuremote-native.service`. @@ -78,9 +78,11 @@ Scaled GNOME desktops using native Portal input use EIS on the same authorized s GNOME Portal CONTROL no longer waits for a consent dialog or the first video frame. It announces the selected desktop before reporting secure-desktop state and receiving the official viewer's “auto unlock host” request. The official Android viewer requires this desktop selection before entering its auto-unlock flow, so the notification must precede capture readiness; actual screen dimensions, codec information and capture permissions still follow their real readiness state. Enable auto unlock in the viewer and use the current Linux user's login password. The host verifies it with PAM, unlocks the existing GNOME session, requests display wake, and restores capture, input and clipboard through the shared Portal session. Initial consent or revoked permissions still require local approval. Reconnect if physical or logical dimensions change. -The host also accepts a desktop connection when the service started on an unlocked desktop (no Portal yet) and the screen was locked afterwards (a cold connection): it reports the lock state and accepts auto unlock, then restores picture, input and clipboard after unlock. Verified on Ubuntu GNOME Wayland with the official Mac viewer; the official iPhone 4.42.0 does not send the auto-unlock request in this scenario and is unresolved ([#29](../../issues/29)). **Limitation**: without 「自动解锁被控端」 (auto unlock) enabled, the locked screen stays black and the unlock page cannot be activated; in the UU client use 控制中心 → 安全 → 自动解锁被控端 and enter the password to unlock. With no saved Portal permission GNOME refuses to create a session while locked, so the picture only appears after unlock. +The host also accepts a desktop connection while GNOME is locked with no Portal session yet (a cold connection): it reports lock state and accepts auto unlock, then restores picture, input and clipboard after unlock. While Portal is pending, the host sends generated black video once per second on the negotiated track by default, allowing the official iPhone viewer to enter its auto-unlock flow without extra configuration. Verified on Ubuntu GNOME Wayland with the official Mac viewer; iPhone 4.42.0 with H.265 completed a same-executable control experiment: black video ON succeeded, OFF failed, and ON succeeded again ([#29](../../issues/29)). **Limitation**: this is synthetic black video, not the GNOME lock screen, and it does not enable native mouse/keyboard input while locked. Enable 「自动解锁被控端」 in the UU client under 控制中心 → 安全 and enter the password. Without saved Portal permission, local consent is still required after unlock before the real desktop can appear. -The five-minute screen-off and lock settings remain unchanged. Only capture recovery in an unlocked session temporarily holds a GNOME idle inhibitor; the first frame, lock, disconnect or initialization failure releases it. An idle viewer still allows the normal screen-off and lock timeout, and manual lock remains effective. Desktop input is blocked and video is hidden while locked or when lock state is unknown. Independent terminal, file-transfer and port-mapping peers do not wait for video initialization and survive desktop reconnection. Automated protocol and lifecycle regression tests cover this path; auto unlock and capture recovery after a cold locked connection are verified on GNOME hardware with the official Mac viewer, while screen-off and the other scenarios still need acceptance testing. Waking a fully suspended or hibernating machine is outside this CONTROL path. +Existing screen-off and lock settings remain unchanged. Only capture recovery in an unlocked session temporarily holds a GNOME idle inhibitor; the first frame, lock, disconnect or initialization failure releases it. An idle viewer still allows the normal screen-off and lock timeout, and manual lock remains effective. Desktop input is blocked and real video is hidden while locked or when lock state is unknown. Independent terminal, file-transfer and port-mapping peers do not wait for video initialization and survive desktop reconnection. Automated protocol and lifecycle regression tests cover this path; auto unlock and capture recovery after a cold locked connection are verified on GNOME hardware with the official Mac and iPhone 4.42.0 viewers, while screen-off and the other scenarios still need acceptance testing. Waking a fully suspended or hibernating machine is outside this CONTROL path. + +Locked black video requires FFmpeg's libx264/libx265 encoder for the negotiated H.264/H.265 codec; hardware encoding support alone does not satisfy this dependency. Encoder failures are logged explicitly. Synthetic video does not change capture permissions or advertised monitor dimensions. It pauses on unlock or unknown lock state, ends when capture is ready/rejected or the connection closes, and then hands the same video track to real capture. See the [experiment record](docs/experiments/issue-29-locked-video.md) for the controlled timeline and validation scope. The native Wayland backend needs `wlr-output-power-management` from the compositor to wake a turned-off screen; without it only wake is unavailable — capture and encoding are unaffected. diff --git a/README.md b/README.md index 56f0fcf..49037f5 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ Portal 仅作用于本服务获授权的会话。文件传输、终端等独立 | Hyprland | Wayland（wlroots 原生协议） | 🧪 实验性，虚拟机测试（部分功能受限，见下） | `--features wayland` | | GNOME / KDE Plasma | Wayland（xdg-desktop-portal） | 🧪 实验性，验证范围与限制见下 | `--features wayland,portal` | -GNOME Wayland 的 Portal/EIS 路径已在 Ubuntu 26.04、GNOME 50、单屏 4K/200% 缩放下验证画面和键鼠；锁屏冷连接自动解锁已用 Mac 客户端实机验证，其余锁屏恢复及重连仍待验收。除 GNOME 的上述路径和 X11 外，其余桌面目前主要在虚拟机里测试。如果你在真机上遇到问题，欢迎[提交 issue](../../issues)， +GNOME Wayland 的 Portal/EIS 路径已在 Ubuntu 26.04、GNOME 50、单屏 4K/200% 缩放下验证画面和键鼠；锁屏冷连接自动解锁已用 Mac 和 iPhone 4.42.0 客户端实机验证，其余锁屏恢复及重连仍待验收。除 GNOME 的上述路径和 X11 外，其余桌面目前主要在虚拟机里测试。如果你在真机上遇到问题，欢迎[提交 issue](../../issues)， 请附上发行版、桌面环境、显卡型号和 `journalctl --user -u uuremote-native.service` 里的相关日志。 Wayland 下的限制：只支持单屏、固定分辨率；Hyprland 原生后端默认使用双缓冲流水线 shm screencopy，控制端帧率设置会同步到采集节拍和编码器，但实际帧率仍受合成器、渲染和读回速度限制（旧串行路径在 2560x1600 曾测得约 16～22 帧；流水线只在虚拟机里测量过，不代表实机保证）；portal 后端暂不支持动态分辨率；GNOME 的屏幕唤醒与锁屏恢复见下；剪贴板只在控制端的远控窗口处于前台时同步。 @@ -70,9 +70,11 @@ GNOME 缩放桌面在使用 Portal 原生键鼠时，若支持 `ConnectToEIS` GNOME Portal 的控制通道不再等待授权窗口或第一帧画面：先通知客户端当前选择桌面，再上报锁屏状态、接收官方客户端的「自动解锁被控端」请求。官方 Android 客户端需要桌面目标通知才会进入自动解锁流程，因此该通知不能等待采集就绪；真实屏幕尺寸、编码信息及采集权限仍按实际就绪状态上报。请在客户端启用自动解锁，使用当前 Linux 用户的登录密码；主机通过 PAM 验证后解锁当前 GNOME 会话、请求点亮屏幕，并恢复同一 Portal 会话中的画面、键鼠和剪贴板。首次授权或权限被撤销时仍需本机确认。物理或逻辑分辨率变化后需要重连。 -服务启动时桌面已解锁、尚未建立 Portal，随后锁屏再连接（冷连接）时，主机也会建立控制通道：上报锁屏状态并接受自动解锁，解锁后再恢复画面、键鼠和剪贴板。官方 Mac 客户端已在 Ubuntu GNOME Wayland 上验证；官方 iPhone 4.42.0 在该场景下不发送自动解锁请求，尚未解决（[#29](../../issues/29)）。**局限**：未开启「自动解锁被控端」时，锁屏下仍是黑屏，无法唤起解锁页；请在 UU 客户端 控制中心 → 安全 → 自动解锁被控端 中输入密码解锁。若本机没有已保存的 Portal 授权，GNOME 在锁屏时不允许创建会话，画面要等解锁后才出现。 +桌面锁屏且尚未建立 Portal 时（冷连接），主机也会建立控制通道：上报锁屏状态并接受自动解锁，解锁后再恢复画面、键鼠和剪贴板。Portal 等待就绪期间，主机默认通过已协商的视频轨每秒发送一帧生成的纯黑画面，让官方 iPhone 客户端进入自动解锁流程，无需额外配置。Ubuntu GNOME Wayland 上已用官方 Mac 客户端验证；iPhone 4.42.0 的 H.265 冷连接完成了同一实验二进制「开启黑帧成功 → 关闭失败 → 再开成功」对照（[#29](../../issues/29)）。**局限**：黑帧不是真实 GNOME 锁屏画面，也不支持在锁屏界面操作原生键鼠；需要在 UU 客户端 控制中心 → 安全 → 自动解锁被控端 中启用自动解锁并输入密码。若本机没有已保存的 Portal 授权，解锁后仍需本机确认授权才能显示真实桌面。 -无需修改五分钟熄屏和锁屏设置。主机仅在会话已解锁的采集恢复期间临时申请 GNOME 的 idle 抑制，拿到首帧后立即释放；锁屏、断开或初始化失败也会释放。远控窗口空闲时仍按原策略熄屏和锁屏，手动锁屏也继续生效。锁屏及锁状态未知时停止桌面输入并隐藏画面；独立终端、文件传输和端口映射连接不依赖画面初始化，也不会因桌面重连被关闭。这一恢复路径已覆盖自动化协议与生命周期回归测试，锁屏冷连接后的自动解锁与采集恢复已用官方 Mac 客户端在 GNOME 实机验证，熄屏等其余场景仍待验收；不支持从整机挂起或休眠状态通过现有控制通道唤醒。 +无需修改现有熄屏和锁屏设置。主机仅在会话已解锁的采集恢复期间临时申请 GNOME 的 idle 抑制，拿到首帧后立即释放；锁屏、断开或初始化失败也会释放。远控窗口空闲时仍按原策略熄屏和锁屏，手动锁屏也继续生效。锁屏及锁状态未知时停止桌面输入并隐藏真实画面；独立终端、文件传输和端口映射连接不依赖画面初始化，也不会因桌面重连被关闭。这一恢复路径已覆盖自动化协议与生命周期回归测试，锁屏冷连接后的自动解锁与采集恢复已用官方 Mac 和 iPhone 4.42.0 客户端在 GNOME 实机验证，熄屏等其余场景仍待验收；不支持从整机挂起或休眠状态通过现有控制通道唤醒。 + +锁屏黑帧使用 FFmpeg 的 libx264/libx265 编码器（按已协商的 H.264/H.265 选择），硬件编码器可用并不代表这些软件编码器已安装。编码失败会明确记录日志。黑帧不改变采集权限或上报的真实屏幕尺寸；解锁或锁状态未知时暂停，采集就绪、拒绝或连接关闭时结束，再由真实视频接续同一条视频轨。对照时间线与验证范围见[实验记录](docs/experiments/issue-29-locked-video.md)。 原生 Wayland 后端需要合成器提供 `wlr-output-power-management` 才能唤醒已关闭的屏幕；缺少该协议不影响采集和编码。 diff --git a/docs/experiments/issue-29-locked-video.md b/docs/experiments/issue-29-locked-video.md new file mode 100644 index 0000000..36d1644 --- /dev/null +++ b/docs/experiments/issue-29-locked-video.md @@ -0,0 +1,98 @@ +# Issue 29: iPhone cold locked auto unlock + +## Observed failure and controlled result + +On a cold locked GNOME Portal connection, the official iPhone 4.42.0 viewer +completed CONTROL Echo/queries and received secure-desktop state, but did not +send SimpleAction 15. Replaying secure state and announcing CaptureChange early +were insufficient. The experiment varied synthetic video alone; actual pending +capture permissions remained Unknown, with no extra screen/codec initialization. + +All times below are October 17, 2026, Asia/Taipei (UTC+08:00). All three trials +used the identical experimental executable, SHA-256 +`11b4de0328ca6b4b656d225c1de37f95cdcb104f0ef458b76e2444bcc3102663`. +Only its temporary `UU_LOCKED_VIDEO_PROBE` flag changed. Each trial restarted the +service while GNOME was locked, with iPhone as the first desktop viewer. + +| Trial | Black video | Safe host evidence | Human result | +|---|---|---|---| +| First cold ON | Enabled | H.265 frame completed 11:13:55.160912; action 15 at 11:13:55.204820; unlocked 11:13:55.786608; auto unlock Success 11:13:55.819243 | Real desktop recovered | +| Cold OFF | Disabled; restarted 11:17:52.880738 | Connected 11:18:04.342190; no synthetic frame, action 15 or unlock result; closed 11:18:12.439870 | Failed | +| Independent cold ON repeat | Enabled; restarted 11:19:08.293631 | First H.265 frame 11:19:14.108291; action 15 at 11:19:15.868938; auto unlock Success 11:19:16.682968; unlocked 11:19:16.735692; real initialization 11:19:16.993977 | Real desktop recovered | + +The first trial sent one synthetic frame, followed by 857 real frames and growing +receiver sequence reports. The independent repeat sent three synthetic frames. +An additional successful connection at 11:16 had prior Portal history and is not +counted as a cold trial. The OFF trial's unlock at 11:18:16.471908 occurred after +closure and is not remote-unlock success. + +The ON/OFF/ON result supports missing pending-locked video as the actionable host +dependency for this client. It does not establish the iOS implementation, prove +that one frame is always sufficient, or justify invented permission claims. + +## Default implementation + +`src/locked_video.rs` generates one independently decodable 1280x720 black IDR +using configured FFmpeg and libx264/libx265 for the negotiated H.264/H.265 codec. +Hardware encoding availability alone does not guarantee these software encoders; +encoding failure is reported. Generation has a five-second timeout and kills the +child on cancellation. The encoded access unit is cached for the desktop peer. + +When Portal is pending, lock state is exactly locked and the peer is connected, +the host sends the frame once per second on the existing negotiated video track, +with capture index 0 and new-frame RTP metadata. It shares the real video's SSRC, +sequence and timestamp timeline. Logs count completed sends through the track's +transmission revision; a paused no-op write is not counted as transmitted video. +Each send is bounded to two seconds, and pacing resets after it completes. + +The 1280x720 size belongs only to the synthetic bitstream. Screen inventory, +codec CONTROL initialization, actual permissions, input and clipboard gating +continue to follow Portal readiness. Unlock or unknown lock state pauses synthetic +video; capture ready/rejected or peer/SCTP closure ends it. Real streaming starts +after this task returns, so synthetic and live video do not write concurrently. +There is no experimental flag or attempt cap in the default implementation. + +The frame is not a capture of GNOME's lock screen and cannot enable native input +there. The viewer still needs auto unlock enabled and the Linux login password. +Existing GNOME screen-off/lock policy and independent tool peers remain intact. + +## Validation and limits + +Codec tests decode H.264 and H.265 output into exactly 1280x720 black pixels. +Virtual-network WebRTC tests exercise suppression before connect and while +unlocked/unknown, pending-locked transmission, ready/rejected/closed lifecycle, +and transition to live video on the same RTP timeline. These tests require local +FFmpeg and are explicitly run despite being marked ignored in the normal suite. + +The physical ON/OFF/ON acceptance above covers the experimental binary on this +GNOME host, iPhone 4.42.0 and H.265. The default implementation was then installed +and restarted while locked, with no experimental flag. Its installed and running +executable hashes both matched SHA-256 +`576cf6a01d108c0311fb8139a85ad8fe582c1d43324c9b643246d394963fd379`. +The first desktop connection sent black video at 11:26:13.701815, received +action 15 at 11:26:17.707857 and reported auto unlock Success at +11:26:18.420772. Lock state became unlocked at 11:26:19.247680; synthetic video +ended after six completed frames, followed by real initialization at +11:26:19.342018 and receiver sequence growth. The user also confirmed that the +phone displayed the desktop normally on this final executable, completing +cold-connection acceptance of the default unlock and video handoff path. + +Before submission, review identified a race when unlock or Portal readiness +arrived in the encoder's final poll. The source now prioritizes state-change +notifications and rechecks current Portal, lock and connection state before +encoding and before sending. A deterministic virtual-network regression publishes +unlock, unknown lock state or readiness during that final encoder poll and checks +that no synthetic RTP is emitted. This review follow-up is covered by automated +tests; the executable hashes and phone acceptance above describe the preceding +installed build, not a second physical acceptance of this follow-up. + +H.264-only iPhone connections, other client versions, prolonged pending sessions, +other host/encoder combinations and slower WAN/relay paths are not established +by these trials. Full suspend or hibernate wake is outside this CONTROL path. + +For future acceptance, preserve the executable hash and relevant settings, restart +while actually locked, keep iPhone as the first desktop viewer, and correlate +safe frame counts, action 15, unlock result, lock transition, Portal recovery and +real receiver reports. Do not count manual unlock or a Mac warm-up. RTP reception +alone does not prove phone decoding or visible desktop; obtain a human result. +Never export passwords, tokens, QR URLs or user content with diagnostic evidence. diff --git a/src/locked_video.rs b/src/locked_video.rs new file mode 100644 index 0000000..951d441 --- /dev/null +++ b/src/locked_video.rs @@ -0,0 +1,101 @@ +//! Black video keeps locked Portal initialization reachable by the iPhone viewer. +use std::time::Duration; +use bytes::Bytes; +use webrtc::{media::Sample, peer_connection::{RTCPeerConnection, peer_connection_state::RTCPeerConnectionState}}; +use crate::{capture_setting::VideoCodec, desktop_session::PortalDesktop, video_track::VideoTrack}; + +async fn black_frame(codec: VideoCodec) -> Result { + let mut command = crate::video::ffmpeg_command(); + command.args(["-hide_banner", "-loglevel", "error", "-nostdin", + "-f", "lavfi", "-i", "color=c=black:s=1280x720:r=1", "-frames:v", "1", + "-an", "-pix_fmt", "yuv420p", "-threads", "1", "-preset", "ultrafast", "-tune", "zerolatency"]); + match codec { + VideoCodec::H264 => { command.args(["-c:v", "libx264", "-profile:v", "baseline", + "-x264-params", "keyint=1:repeat-headers=1:aud=1", "-f", "h264"]); }, + VideoCodec::H265 => { command.args(["-c:v", "libx265", + "-x265-params", "keyint=1:repeat-headers=1:aud=1:pools=1:frame-threads=1:log-level=error", "-f", "hevc"]); }, + } + command.arg("pipe:1").kill_on_drop(true); + let output = tokio::time::timeout(Duration::from_secs(5), command.output()).await + .map_err(|_| "Locked desktop video encoder timed out".to_string())? + .map_err(|error| format!("Locked desktop video encoder: {error}"))?; + if !output.status.success() || output.stdout.is_empty() { + return Err(format!("Locked desktop video encoder failed: status={}", output.status)); + } + Ok(output.stdout.into()) +} + +pub async fn run(pc: &RTCPeerConnection, track: &VideoTrack, desktop: &PortalDesktop) -> Result<(), String> { + run_with_encoder(pc, track, desktop, black_frame).await +} + +async fn run_with_encoder(pc: &RTCPeerConnection, track: &VideoTrack, desktop: &PortalDesktop, mut encode: F) -> Result<(), String> +where F: FnMut(VideoCodec) -> Fut, Fut: std::future::Future> { + // #29: iPhone 4.42.0 cold ON/OFF/ON tests only sent action15 after black video. + // Keep CONTROL permissions truthful and use the existing video track until real capture is ready. + let mut updates = desktop.subscribe(); + let mut locked = desktop.lock_state(); + let mut tick = tokio::time::interval(Duration::from_secs(1)); + tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + let mut frame = None; + let mut sent = 0u64; + let mut attempts = 0u64; + loop { + let state = updates.borrow_and_update().clone(); + let _ = locked.borrow_and_update(); + if state.ready() || !state.waiting || crate::video::transport_closed(pc) { + if attempts > 0 { log::info!("Locked desktop video ended: sent={sent} attempts={attempts}"); } + return Ok(()); + } + tokio::select! { + biased; + changed = updates.changed() => if changed.is_err() { return Ok(()); }, + changed = locked.changed() => if changed.is_err() { return Ok(()); }, + _ = tick.tick() => { + if !updates.borrow().waiting || *locked.borrow() != Some(true) + || pc.connection_state() != RTCPeerConnectionState::Connected { continue; } + if frame.is_none() { + let codec = crate::capture_setting::preferred_codec( + track.supports(VideoCodec::H264), track.supports(VideoCodec::H265)); + let encoded = tokio::select! { + biased; + _ = updates.changed() => continue, + _ = locked.changed() => continue, + result = encode(codec) => result?, + }; + frame = Some((codec, encoded)); + } + if crate::video::transport_closed(pc) { return Ok(()); } + // Encoding can finish in the same poll that publishes unlock/readiness. + if !updates.borrow().waiting || *locked.borrow() != Some(true) + || pc.connection_state() != RTCPeerConnectionState::Connected { continue; } + let (codec, data) = frame.as_ref().unwrap(); + let sample = Sample { data: data.clone(), duration: Duration::from_secs(1), ..Default::default() }; + let extensions = [crate::rtp_metadata::capture_index(0), crate::rtp_metadata::new_frame()]; + let revision = track.revision(); + // No monitor inventory or permission claims: 1280x720 is only this synthetic bitstream. + tokio::select! { + biased; + _ = updates.changed() => continue, + _ = locked.changed() => continue, + result = tokio::time::timeout(Duration::from_secs(2), + track.write_sample_with_extensions(*codec, &sample, &extensions)) => { + if crate::video::transport_closed(pc) { return Ok(()); } + result.map_err(|_| "Locked desktop video send timed out".to_string())? + .map_err(|error| format!("Locked desktop video send: {error}"))?; + }, + } + attempts += 1; + if track.revision() > revision { sent += 1; } + tick.reset(); + if attempts == 1 || attempts % 10 == 0 { + log::info!("Locked desktop video: sent={sent} codec={codec:?} synthetic_size=1280x720 bytes={} attempts={attempts}", data.len()); + } + }, + } + } +} + +#[cfg(test)] +#[path = "locked_video_tests.rs"] +mod tests; diff --git a/src/locked_video_tests.rs b/src/locked_video_tests.rs new file mode 100644 index 0000000..bbbca3e --- /dev/null +++ b/src/locked_video_tests.rs @@ -0,0 +1,350 @@ +use super::{black_frame, run, run_with_encoder}; +use crate::{ + capture::ScreenCapture, + capture_setting::VideoCodec, + desktop_backend::DesktopBackend, + desktop_session::PortalDesktop, + video_track::VideoTrack, +}; +use std::{process::Stdio, sync::Arc, time::Duration}; +use tokio::{io::AsyncWriteExt, sync::{mpsc, watch}}; +use webrtc::{ + api::{media_engine::MediaEngine, APIBuilder}, + media::Sample, + peer_connection::{ + configuration::RTCConfiguration, + peer_connection_state::RTCPeerConnectionState, + RTCPeerConnection, + }, + rtp::packet::Packet, + rtp_transceiver::{ + rtp_codec::{RTCRtpCodecCapability, RTCRtpCodecParameters, RTPCodecType}, + }, + util::vnet::router::Router, +}; + +async fn decode_gray(data: &[u8], demuxer: &str) -> (Vec, String) { + let mut command = crate::video::ffmpeg_command(); + command.args([ + "-hide_banner", "-loglevel", "info", "-nostdin", "-f", demuxer, "-i", "pipe:0", + "-frames:v", "1", "-vf", "scale=in_range=tv:out_range=pc,format=gray,showinfo", + "-pix_fmt", "gray", "-f", "rawvideo", "pipe:1", + ]); + command.stdin(Stdio::piped()).stdout(Stdio::piped()).stderr(Stdio::piped()).kill_on_drop(true); + let mut child = command.spawn().expect("start FFmpeg decoder"); + let mut stdin = child.stdin.take().expect("FFmpeg decoder stdin"); + let output = tokio::time::timeout(Duration::from_secs(10), async move { + stdin.write_all(data).await.expect("write encoded frame to FFmpeg"); + drop(stdin); + child.wait_with_output().await.expect("wait for FFmpeg decoder") + }).await.expect("FFmpeg decoder timed out"); + assert!(output.status.success(), "FFmpeg decoder failed: status={}", output.status); + (output.stdout, String::from_utf8_lossy(&output.stderr).into_owned()) +} + +async fn black_frame_decodes_as_1280x720_black(codec: VideoCodec, demuxer: &str) { + let encoded = black_frame(codec).await.expect("encode black frame"); + assert!(!encoded.is_empty()); + let (gray, diagnostics) = decode_gray(&encoded, demuxer).await; + assert!(diagnostics.contains("s:1280x720"), "decoded dimensions were not 1280x720"); + assert_eq!(gray.len(), 1280 * 720, "expected one 1280x720 gray frame"); + assert!(gray.iter().all(|&pixel| pixel <= 1), "decoded frame was not black"); +} + +#[tokio::test] +#[ignore = "requires FFmpeg libx264/libx265"] +async fn h264_black_frame_decodes_to_1280x720_black() { + black_frame_decodes_as_1280x720_black(VideoCodec::H264, "h264").await; +} + +#[tokio::test] +#[ignore = "requires FFmpeg libx264/libx265"] +async fn h265_black_frame_decodes_to_1280x720_black() { + black_frame_decodes_as_1280x720_black(VideoCodec::H265, "hevc").await; +} + +fn ready_screen() -> Arc { + let mut screen = ScreenCapture::unavailable("synthetic ready desktop".into()); + screen.backend = DesktopBackend::Portal; + screen.init_error = None; + screen.display = "wayland:test".into(); + screen.width = 1280; + screen.height = 720; + Arc::new(screen) +} + +async fn receive_frame(packets: &mut mpsc::UnboundedReceiver, within: Duration) -> Vec { + tokio::time::timeout(within, async { + let mut frame = Vec::new(); + loop { + let packet = packets.recv().await.expect("RTP packet receiver closed"); + let marker = packet.header.marker; + frame.push(packet); + if marker { + return frame; + } + } + }).await.expect("timed out waiting for an RTP frame") +} + +async fn assert_no_rtp(packets: &mut mpsc::UnboundedReceiver, within: Duration) { + assert!(tokio::time::timeout(within, packets.recv()).await.is_err(), + "locked video task sent RTP while transmission was disabled"); +} + +fn assert_single_rtp_frame(frame: &[Packet]) { + let first = frame.first().expect("RTP frame has packets"); + let last = frame.last().unwrap(); + assert!(last.header.marker, "RTP frame ends with a marker packet"); + for pair in frame.windows(2) { + assert_eq!(pair[0].header.ssrc, pair[1].header.ssrc); + assert_eq!(pair[0].header.timestamp, pair[1].header.timestamp); + assert_eq!(pair[1].header.sequence_number, pair[0].header.sequence_number.wrapping_add(1)); + } + assert!(frame.iter().all(|packet| packet.header.ssrc == first.header.ssrc)); +} + +struct VirtualPeers { + host: Arc, + remote: Arc, + _control: Arc, + router: Arc>, + track: Arc, + packets: mpsc::UnboundedReceiver, +} + +impl VirtualPeers { + async fn new(codec: VideoCodec) -> Self { + let (remote_setting, host_setting, router) = crate::test_network::virtual_network().await; + + let mut remote_media = MediaEngine::default(); + match codec { + VideoCodec::H264 => remote_media.register_codec(RTCRtpCodecParameters { + capability: RTCRtpCodecCapability { + mime_type: VideoCodec::H264.mime().into(), + clock_rate: 90_000, + sdp_fmtp_line: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f".into(), + ..Default::default() + }, + payload_type: 102, + ..Default::default() + }, RTPCodecType::Video).unwrap(), + VideoCodec::H265 => crate::video_track::register_hevc(&mut remote_media).unwrap(), + } + let remote = Arc::new(APIBuilder::new().with_setting_engine(remote_setting) + .with_media_engine(remote_media).build() + .new_peer_connection(RTCConfiguration::default()).await.unwrap()); + remote.add_transceiver_from_kind(RTPCodecType::Video, None).await.unwrap(); + let control = remote.create_data_channel("CONTROL_DATA_CHANNEL", None).await.unwrap(); + + let mut host_media = MediaEngine::default(); + host_media.register_default_codecs().unwrap(); + crate::video_track::register_hevc(&mut host_media).unwrap(); + let host = Arc::new(APIBuilder::new().with_setting_engine(host_setting) + .with_media_engine(host_media).build() + .new_peer_connection(RTCConfiguration::default()).await.unwrap()); + let track = Arc::new(VideoTrack::new()); + host.add_track(track.clone()).await.unwrap(); + + let (packet_tx, packets) = mpsc::unbounded_channel(); + remote.on_track(Box::new(move |remote_track, _, _| { + let packet_tx = packet_tx.clone(); + Box::pin(async move { + tokio::spawn(async move { + while let Ok((packet, _)) = remote_track.read_rtp().await { + if packet_tx.send(packet).is_err() { break; } + } + }); + }) + })); + + Self { host, remote, _control: control, router, track, packets } + } + + async fn connect(&self, codec: VideoCodec) { + let offer = self.remote.create_offer(None).await.unwrap(); + let mut gathered = self.remote.gathering_complete_promise().await; + self.remote.set_local_description(offer).await.unwrap(); + gathered.recv().await; + self.host.set_remote_description(self.remote.local_description().await.unwrap()).await.unwrap(); + let answer = self.host.create_answer(None).await.unwrap(); + let mut gathered = self.host.gathering_complete_promise().await; + self.host.set_local_description(answer).await.unwrap(); + gathered.recv().await; + self.remote.set_remote_description(self.host.local_description().await.unwrap()).await.unwrap(); + tokio::time::timeout(Duration::from_secs(15), async { + while self.host.connection_state() != RTCPeerConnectionState::Connected + || !self.track.supports(codec) + { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }).await.expect("virtual WebRTC peers did not connect with the requested codec"); + assert!(!self.track.supports(match codec { + VideoCodec::H264 => VideoCodec::H265, + VideoCodec::H265 => VideoCodec::H264, + })); + } + + async fn close(self) { + self.host.close().await.unwrap(); + self.remote.close().await.unwrap(); + self.router.lock().await.stop().await.unwrap(); + } +} + +async fn exercise_locked_video(codec: VideoCodec) { + tokio::time::timeout(Duration::from_secs(35), async { + let mut peers = VirtualPeers::new(codec).await; + let (lock_tx, lock_rx) = watch::channel(Some(true)); + let desktop = Arc::new(PortalDesktop::with_lock_state( + Arc::new(ScreenCapture::unavailable("synthetic pending desktop".into())), + lock_rx, + )); + assert!(desktop.snapshot().waiting); + assert!(!desktop.snapshot().ready()); + + let run_host = peers.host.clone(); + let run_track = peers.track.clone(); + let run_desktop = desktop.clone(); + let task = tokio::spawn(async move { run(&run_host, &run_track, &run_desktop).await }); + + assert_ne!(peers.host.connection_state(), RTCPeerConnectionState::Connected); + assert_no_rtp(&mut peers.packets, Duration::from_millis(150)).await; + + peers.connect(codec).await; + + let locked_frame = receive_frame(&mut peers.packets, Duration::from_secs(5)).await; + assert_single_rtp_frame(&locked_frame); + let last_locked_packet = locked_frame.last().unwrap().clone(); + + lock_tx.send_replace(None); + assert_no_rtp(&mut peers.packets, Duration::from_millis(1200)).await; + lock_tx.send_replace(Some(false)); + assert_no_rtp(&mut peers.packets, Duration::from_millis(1200)).await; + + desktop.publish_test_screen(ready_screen()); + tokio::time::timeout(Duration::from_secs(2), task).await + .expect("locked video task did not stop when the desktop became ready") + .expect("locked video task panicked") + .expect("locked video task returned an error"); + + let live_data = black_frame(codec).await.expect("encode live test sample"); + peers.track.write_sample_with_extensions(codec, &Sample { + data: live_data, + duration: Duration::from_millis(20), + ..Default::default() + }, &[]).await.unwrap(); + let live_frame = receive_frame(&mut peers.packets, Duration::from_secs(5)).await; + assert_single_rtp_frame(&live_frame); + let first_live_packet = &live_frame[0]; + assert_eq!(first_live_packet.header.ssrc, last_locked_packet.header.ssrc); + assert_eq!(first_live_packet.header.sequence_number, + last_locked_packet.header.sequence_number.wrapping_add(1)); + assert_eq!(first_live_packet.header.timestamp, + last_locked_packet.header.timestamp.wrapping_add(90_000)); + + peers.close().await; + }).await.expect("locked video WebRTC test exceeded its deadline"); +} + +#[tokio::test] +#[ignore = "requires FFmpeg libx264/libx265"] +async fn locked_video_gates_on_pending_locked_connected_and_preserves_video_timeline() { + exercise_locked_video(VideoCodec::H264).await; + exercise_locked_video(VideoCodec::H265).await; +} + +#[tokio::test] +async fn locked_video_rechecks_state_when_encoding_finishes() { + #[derive(Clone, Copy)] + enum Transition { Unlocked, Unknown, Ready } + tokio::time::timeout(Duration::from_secs(20), async { + let mut peers = VirtualPeers::new(VideoCodec::H264).await; + peers.connect(VideoCodec::H264).await; + for transition in [Transition::Unlocked, Transition::Unknown, Transition::Ready] { + let (lock_tx, lock_rx) = watch::channel(Some(true)); + let desktop = Arc::new(PortalDesktop::with_lock_state( + Arc::new(ScreenCapture::unavailable("synthetic pending desktop".into())), + lock_rx, + )); + let run_host = peers.host.clone(); + let run_track = peers.track.clone(); + let run_desktop = desktop.clone(); + let (encoded_tx, mut encoded_rx) = mpsc::unbounded_channel(); + let revision = peers.track.revision(); + let task = tokio::spawn(async move { + run_with_encoder(&run_host, &run_track, &run_desktop, |_| { + let desktop = run_desktop.clone(); + let lock_tx = lock_tx.clone(); + let encoded_tx = encoded_tx.clone(); + async move { + // Change state during the encoder's final poll, before it returns Ready. + match transition { + Transition::Unlocked => { lock_tx.send_replace(Some(false)); }, + Transition::Unknown => { lock_tx.send_replace(None); }, + Transition::Ready => desktop.publish_test_screen(ready_screen()), + } + encoded_tx.send(()).unwrap(); + Ok(bytes::Bytes::from_static(&[0, 0, 1, 65, 1, 2])) + } + }).await + }); + tokio::time::timeout(Duration::from_secs(2), encoded_rx.recv()).await + .expect("test encoder was not polled").expect("test encoder channel closed"); + assert_no_rtp(&mut peers.packets, Duration::from_millis(250)).await; + assert_eq!(peers.track.revision(), revision, "stale state allowed a synthetic frame"); + desktop.publish_test_screen(ready_screen()); + tokio::time::timeout(Duration::from_secs(2), task).await + .expect("locked video task did not stop after readiness") + .expect("locked video task panicked") + .expect("locked video task returned an error"); + } + peers.close().await; + }).await.expect("locked video concurrent-transition test exceeded its deadline"); +} + +#[tokio::test] +async fn locked_video_stops_on_capture_rejection_and_terminal_peer_close() { + tokio::time::timeout(Duration::from_secs(25), async { + let mut peers = VirtualPeers::new(VideoCodec::H264).await; + peers.connect(VideoCodec::H264).await; + let (_lock_tx, lock_rx) = watch::channel(Some(false)); + + let rejected_desktop = Arc::new(PortalDesktop::with_lock_state( + Arc::new(ScreenCapture::unavailable("synthetic pending desktop".into())), + lock_rx.clone(), + )); + let run_host = peers.host.clone(); + let run_track = peers.track.clone(); + let run_desktop = rejected_desktop.clone(); + let rejection_task = tokio::spawn(async move { run(&run_host, &run_track, &run_desktop).await }); + rejected_desktop.publish_test_screen( + Arc::new(ScreenCapture::unavailable("synthetic capture rejection".into())), + ); + assert_eq!(rejected_desktop.snapshot().permission(true), + crate::system_metrics::Permission::Reject); + tokio::time::timeout(Duration::from_secs(2), rejection_task).await + .expect("locked video task did not stop after capture rejection") + .expect("locked video task panicked") + .expect("locked video task returned an error"); + + let pending_desktop = Arc::new(PortalDesktop::with_lock_state( + Arc::new(ScreenCapture::unavailable("synthetic pending desktop".into())), + lock_rx, + )); + let run_host = peers.host.clone(); + let run_track = peers.track.clone(); + let run_desktop = pending_desktop; + let close_task = tokio::spawn(async move { run(&run_host, &run_track, &run_desktop).await }); + peers.host.close().await.unwrap(); + tokio::time::timeout(Duration::from_secs(2), close_task).await + .expect("locked video task did not stop after terminal peer close") + .expect("locked video task panicked") + .expect("locked video task returned an error"); + + peers.packets.close(); + assert!(peers.packets.try_recv().is_err(), "locked video task sent RTP while unlocked"); + peers.remote.close().await.unwrap(); + peers.router.lock().await.stop().await.unwrap(); + }).await.expect("locked video lifecycle test exceeded its deadline"); +} diff --git a/src/main.rs b/src/main.rs index 6f80a0b..ecd0b00 100644 --- a/src/main.rs +++ b/src/main.rs @@ -27,6 +27,7 @@ mod terminal; mod webrtc; mod video; mod video_track; +mod locked_video; #[cfg(test)] mod test_network; mod hevc; diff --git a/src/video.rs b/src/video.rs index 7277a42..e69024d 100644 --- a/src/video.rs +++ b/src/video.rs @@ -274,7 +274,7 @@ fn ffmpeg_spec() -> FfmpegSpec { } } -fn ffmpeg_command() -> Command { ffmpeg_spec().command() } +pub(crate) fn ffmpeg_command() -> Command { ffmpeg_spec().command() } #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum FfmpegTimingMode { Modern, Legacy } @@ -673,7 +673,7 @@ fn wayland_capture_command(executable:&std::path::Path,fps:u32)->Command { command } -fn transport_closed(pc: &RTCPeerConnection) -> bool { +pub(crate) fn transport_closed(pc: &RTCPeerConnection) -> bool { matches!(pc.connection_state(), RTCPeerConnectionState::Closed | RTCPeerConnectionState::Failed) || pc.sctp().state() == webrtc::sctp_transport::sctp_transport_state::RTCSctpTransportState::Closed } diff --git a/src/webrtc.rs b/src/webrtc.rs index 9a7a6e5..90c6ec6 100644 --- a/src/webrtc.rs +++ b/src/webrtc.rs @@ -287,12 +287,15 @@ impl WebRtcSession { let state_pc = streaming_pc.clone(); tokio::spawn(async move { let screen = if let Some(desktop) = stream_desktop { + if let Err(error) = crate::locked_video::run(&streaming_pc, &video_track, &desktop).await { + log::warn!("Locked desktop video unavailable: {error}"); + } let mut updates = desktop.subscribe(); loop { let state = updates.borrow_and_update().clone(); if state.ready() { break Some(state.screen); } if !state.waiting { return; } - if matches!(streaming_pc.connection_state(), RTCPeerConnectionState::Closed | RTCPeerConnectionState::Failed) { return; } + if crate::video::transport_closed(&streaming_pc) { return; } tokio::select! { changed = updates.changed() => if changed.is_err() { return; }, _ = tokio::time::sleep(std::time::Duration::from_millis(100)) => {}, From b12d1c76c9085d7fa16d76f038a91b742e396cc4 Mon Sep 17 00:00:00 2001 From: VallierDev <24893073+VallierDev@users.noreply.github.com> Date: Sat, 17 Oct 2026 11:42:20 +0800 Subject: [PATCH 2/2] docs: keep the #29 experiment record in the PR instead of the repository Co-Authored-By: Claude Opus 5.5 --- README.en.md | 2 +- README.md | 2 +- docs/experiments/issue-29-locked-video.md | 98 ----------------------- 3 files changed, 2 insertions(+), 100 deletions(-) delete mode 100644 docs/experiments/issue-29-locked-video.md diff --git a/README.en.md b/README.en.md index bef0226..3209316 100644 --- a/README.en.md +++ b/README.en.md @@ -82,7 +82,7 @@ The host also accepts a desktop connection while GNOME is locked with no Portal Existing screen-off and lock settings remain unchanged. Only capture recovery in an unlocked session temporarily holds a GNOME idle inhibitor; the first frame, lock, disconnect or initialization failure releases it. An idle viewer still allows the normal screen-off and lock timeout, and manual lock remains effective. Desktop input is blocked and real video is hidden while locked or when lock state is unknown. Independent terminal, file-transfer and port-mapping peers do not wait for video initialization and survive desktop reconnection. Automated protocol and lifecycle regression tests cover this path; auto unlock and capture recovery after a cold locked connection are verified on GNOME hardware with the official Mac and iPhone 4.42.0 viewers, while screen-off and the other scenarios still need acceptance testing. Waking a fully suspended or hibernating machine is outside this CONTROL path. -Locked black video requires FFmpeg's libx264/libx265 encoder for the negotiated H.264/H.265 codec; hardware encoding support alone does not satisfy this dependency. Encoder failures are logged explicitly. Synthetic video does not change capture permissions or advertised monitor dimensions. It pauses on unlock or unknown lock state, ends when capture is ready/rejected or the connection closes, and then hands the same video track to real capture. See the [experiment record](docs/experiments/issue-29-locked-video.md) for the controlled timeline and validation scope. +Locked black video requires FFmpeg's libx264/libx265 encoder for the negotiated H.264/H.265 codec; hardware encoding support alone does not satisfy this dependency. Encoder failures are logged explicitly. Synthetic video does not change capture permissions or advertised monitor dimensions. It pauses on unlock or unknown lock state, ends when capture is ready/rejected or the connection closes, and then hands the same video track to real capture. See [#29](../../issues/29) and [#34](../../pull/34) for the controlled experiment and validation scope. The native Wayland backend needs `wlr-output-power-management` from the compositor to wake a turned-off screen; without it only wake is unavailable — capture and encoding are unaffected. diff --git a/README.md b/README.md index 49037f5..06ef0a2 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ GNOME Portal 的控制通道不再等待授权窗口或第一帧画面：先通 无需修改现有熄屏和锁屏设置。主机仅在会话已解锁的采集恢复期间临时申请 GNOME 的 idle 抑制，拿到首帧后立即释放；锁屏、断开或初始化失败也会释放。远控窗口空闲时仍按原策略熄屏和锁屏，手动锁屏也继续生效。锁屏及锁状态未知时停止桌面输入并隐藏真实画面；独立终端、文件传输和端口映射连接不依赖画面初始化，也不会因桌面重连被关闭。这一恢复路径已覆盖自动化协议与生命周期回归测试，锁屏冷连接后的自动解锁与采集恢复已用官方 Mac 和 iPhone 4.42.0 客户端在 GNOME 实机验证，熄屏等其余场景仍待验收；不支持从整机挂起或休眠状态通过现有控制通道唤醒。 -锁屏黑帧使用 FFmpeg 的 libx264/libx265 编码器（按已协商的 H.264/H.265 选择），硬件编码器可用并不代表这些软件编码器已安装。编码失败会明确记录日志。黑帧不改变采集权限或上报的真实屏幕尺寸；解锁或锁状态未知时暂停，采集就绪、拒绝或连接关闭时结束，再由真实视频接续同一条视频轨。对照时间线与验证范围见[实验记录](docs/experiments/issue-29-locked-video.md)。 +锁屏黑帧使用 FFmpeg 的 libx264/libx265 编码器（按已协商的 H.264/H.265 选择），硬件编码器可用并不代表这些软件编码器已安装。编码失败会明确记录日志。黑帧不改变采集权限或上报的真实屏幕尺寸；解锁或锁状态未知时暂停，采集就绪、拒绝或连接关闭时结束，再由真实视频接续同一条视频轨。对照实验与验证范围见 [#29](../../issues/29)、[#34](../../pull/34)。 原生 Wayland 后端需要合成器提供 `wlr-output-power-management` 才能唤醒已关闭的屏幕；缺少该协议不影响采集和编码。 diff --git a/docs/experiments/issue-29-locked-video.md b/docs/experiments/issue-29-locked-video.md deleted file mode 100644 index 36d1644..0000000 --- a/docs/experiments/issue-29-locked-video.md +++ /dev/null @@ -1,98 +0,0 @@ -# Issue 29: iPhone cold locked auto unlock - -## Observed failure and controlled result - -On a cold locked GNOME Portal connection, the official iPhone 4.42.0 viewer -completed CONTROL Echo/queries and received secure-desktop state, but did not -send SimpleAction 15. Replaying secure state and announcing CaptureChange early -were insufficient. The experiment varied synthetic video alone; actual pending -capture permissions remained Unknown, with no extra screen/codec initialization. - -All times below are October 17, 2026, Asia/Taipei (UTC+08:00). All three trials -used the identical experimental executable, SHA-256 -`11b4de0328ca6b4b656d225c1de37f95cdcb104f0ef458b76e2444bcc3102663`. -Only its temporary `UU_LOCKED_VIDEO_PROBE` flag changed. Each trial restarted the -service while GNOME was locked, with iPhone as the first desktop viewer. - -| Trial | Black video | Safe host evidence | Human result | -|---|---|---|---| -| First cold ON | Enabled | H.265 frame completed 11:13:55.160912; action 15 at 11:13:55.204820; unlocked 11:13:55.786608; auto unlock Success 11:13:55.819243 | Real desktop recovered | -| Cold OFF | Disabled; restarted 11:17:52.880738 | Connected 11:18:04.342190; no synthetic frame, action 15 or unlock result; closed 11:18:12.439870 | Failed | -| Independent cold ON repeat | Enabled; restarted 11:19:08.293631 | First H.265 frame 11:19:14.108291; action 15 at 11:19:15.868938; auto unlock Success 11:19:16.682968; unlocked 11:19:16.735692; real initialization 11:19:16.993977 | Real desktop recovered | - -The first trial sent one synthetic frame, followed by 857 real frames and growing -receiver sequence reports. The independent repeat sent three synthetic frames. -An additional successful connection at 11:16 had prior Portal history and is not -counted as a cold trial. The OFF trial's unlock at 11:18:16.471908 occurred after -closure and is not remote-unlock success. - -The ON/OFF/ON result supports missing pending-locked video as the actionable host -dependency for this client. It does not establish the iOS implementation, prove -that one frame is always sufficient, or justify invented permission claims. - -## Default implementation - -`src/locked_video.rs` generates one independently decodable 1280x720 black IDR -using configured FFmpeg and libx264/libx265 for the negotiated H.264/H.265 codec. -Hardware encoding availability alone does not guarantee these software encoders; -encoding failure is reported. Generation has a five-second timeout and kills the -child on cancellation. The encoded access unit is cached for the desktop peer. - -When Portal is pending, lock state is exactly locked and the peer is connected, -the host sends the frame once per second on the existing negotiated video track, -with capture index 0 and new-frame RTP metadata. It shares the real video's SSRC, -sequence and timestamp timeline. Logs count completed sends through the track's -transmission revision; a paused no-op write is not counted as transmitted video. -Each send is bounded to two seconds, and pacing resets after it completes. - -The 1280x720 size belongs only to the synthetic bitstream. Screen inventory, -codec CONTROL initialization, actual permissions, input and clipboard gating -continue to follow Portal readiness. Unlock or unknown lock state pauses synthetic -video; capture ready/rejected or peer/SCTP closure ends it. Real streaming starts -after this task returns, so synthetic and live video do not write concurrently. -There is no experimental flag or attempt cap in the default implementation. - -The frame is not a capture of GNOME's lock screen and cannot enable native input -there. The viewer still needs auto unlock enabled and the Linux login password. -Existing GNOME screen-off/lock policy and independent tool peers remain intact. - -## Validation and limits - -Codec tests decode H.264 and H.265 output into exactly 1280x720 black pixels. -Virtual-network WebRTC tests exercise suppression before connect and while -unlocked/unknown, pending-locked transmission, ready/rejected/closed lifecycle, -and transition to live video on the same RTP timeline. These tests require local -FFmpeg and are explicitly run despite being marked ignored in the normal suite. - -The physical ON/OFF/ON acceptance above covers the experimental binary on this -GNOME host, iPhone 4.42.0 and H.265. The default implementation was then installed -and restarted while locked, with no experimental flag. Its installed and running -executable hashes both matched SHA-256 -`576cf6a01d108c0311fb8139a85ad8fe582c1d43324c9b643246d394963fd379`. -The first desktop connection sent black video at 11:26:13.701815, received -action 15 at 11:26:17.707857 and reported auto unlock Success at -11:26:18.420772. Lock state became unlocked at 11:26:19.247680; synthetic video -ended after six completed frames, followed by real initialization at -11:26:19.342018 and receiver sequence growth. The user also confirmed that the -phone displayed the desktop normally on this final executable, completing -cold-connection acceptance of the default unlock and video handoff path. - -Before submission, review identified a race when unlock or Portal readiness -arrived in the encoder's final poll. The source now prioritizes state-change -notifications and rechecks current Portal, lock and connection state before -encoding and before sending. A deterministic virtual-network regression publishes -unlock, unknown lock state or readiness during that final encoder poll and checks -that no synthetic RTP is emitted. This review follow-up is covered by automated -tests; the executable hashes and phone acceptance above describe the preceding -installed build, not a second physical acceptance of this follow-up. - -H.264-only iPhone connections, other client versions, prolonged pending sessions, -other host/encoder combinations and slower WAN/relay paths are not established -by these trials. Full suspend or hibernate wake is outside this CONTROL path. - -For future acceptance, preserve the executable hash and relevant settings, restart -while actually locked, keep iPhone as the first desktop viewer, and correlate -safe frame counts, action 15, unlock result, lock transition, Portal recovery and -real receiver reports. Do not count manual unlock or a Mac warm-up. RTP reception -alone does not prove phone decoding or visible desktop; obtain a human result. -Never export passwords, tokens, QR URLs or user content with diagnostic evidence.