From b18a77e79ab97b06fb016a4a54567597c8448c9f Mon Sep 17 00:00:00 2001 From: lucas Date: Sat, 17 Oct 2026 13:45:25 +0800 Subject: [PATCH 1/2] fix(portal): recover capture sooner after GNOME display changes Use Mutter notifications to enable a 200 ms grace for correlated, continuously paused capture without a valid format. Cancel stale hints on healthy renegotiation and retain the five-second fallback. Test event ordering, repeated and stale notifications, natural recovery, and subscription teardown. Document measured recovery gains and limits. Fixes #27 --- README.en.md | 2 +- README.md | 2 +- c_engine/portal_engine.c | 42 ++++- docs/experiments/issue-27-display-recovery.md | 82 ++++++++++ tests/portal_engine.c | 145 ++++++++++++++++++ 5 files changed, 265 insertions(+), 8 deletions(-) create mode 100644 docs/experiments/issue-27-display-recovery.md diff --git a/README.en.md b/README.en.md index 3209316..e8f1eb8 100644 --- a/README.en.md +++ b/README.en.md @@ -74,7 +74,7 @@ real hardware, please [open an issue](../../issues) with your distribution, desk Wayland limits: single screen, fixed resolution. The native Hyprland backend now pipelines two shm screencopy buffers and passes the viewer's requested fps to both capture pacing and the encoder. Actual fps remains limited by the compositor, rendering and readback (the old serial path measured about 16–22 fps at 2560x1600; the pipeline is VM-measured only, not a hardware guarantee). The portal backend does not support dynamic resolution; see below for GNOME wake and lock recovery; the clipboard syncs while the viewer's remote window has focus. On the first connection the desktop asks for "Remote Desktop / Screen Sharing" permission; the permission is remembered once granted. -Scaled GNOME desktops using native Portal input use EIS on the same authorized session when `ConnectToEIS` and a screen `mapping_id` are available. Each axis is mapped using the matched region and actual frame dimensions, with no fixed resolution or scale factor. This avoids scaled-coordinate rejection by `NotifyPointerMotionAbsolute`. Portal builds require libei >= 1.1. Missing EIS support retains the legacy input path and its existing scaling limitations; other initialization failures are reported and trigger session recovery. After a successful EIS connection, all input uses EIS. Capture alone and uinput do not initialize EIS; missing optional logical dimensions do not block capture. Keyboard input uses the existing Windows VK to evdev mapping. Recovery keeps the encoder frame dimensions fixed but accepts logical-size changes after scaling. A previously streaming Portal capture that stays paused without a negotiated format for five seconds enters session recovery; a static desktop does not trigger this timeout. Tested while connected on Ubuntu 26.04, GNOME Wayland, one 3840x2160 display: 200% -> 166.67% -> 200%, with continuously updating video and full-screen pointer control after recovery. Switching briefly freezes and goes black before recovering; five seconds is the detection threshold, not the total recovery time. Lock recovery still needs hardware acceptance testing; multi-monitor support, live resolution changes, and rotation/reflection transforms are outside this fix's scope. Capture settings remain at protocol version 4: advertised resolutions use pixel coordinates and DPR 1; Portal logical dimensions are used only for local input mapping. +Scaled GNOME desktops using native Portal input use EIS on the same authorized session when `ConnectToEIS` and a screen `mapping_id` are available. Each axis is mapped using the matched region and actual frame dimensions, with no fixed resolution or scale factor. This avoids scaled-coordinate rejection by `NotifyPointerMotionAbsolute`. Portal builds require libei >= 1.1. Missing EIS support retains the legacy input path and its existing scaling limitations; other initialization failures are reported and trigger session recovery. After a successful EIS connection, all input uses EIS. Capture alone and uinput do not initialize EIS; missing optional logical dimensions do not block capture. Keyboard input uses the existing Windows VK to evdev mapping. Recovery keeps the encoder frame dimensions fixed but accepts logical-size changes after scaling. On GNOME, a display-change notification enables earlier recovery when previously streaming Portal capture remains paused without a valid format for a 200 ms grace after both conditions are present. Healthy renegotiation cancels the early check; repeated notifications do not postpone the same interruption. The five-second fallback remains without a related notification. Static video and valid-format pauses do not trigger recovery. Tested while connected on Ubuntu 26.04, GNOME Wayland, one 3840x2160 display: 200% -> 166.67% -> 200%, with continuously updating video and full-screen pointer control after recovery. Switching briefly freezes and goes black before recovering; five seconds is the fallback threshold, not total recovery time. See the [Issue #27 experiment](docs/experiments/issue-27-display-recovery.md) for timing measurements and validation limits. Lock recovery still needs hardware acceptance testing; multi-monitor support, live resolution changes, and rotation/reflection transforms are outside this fix's scope. Capture settings remain at protocol version 4: advertised resolutions use pixel coordinates and DPR 1; Portal logical dimensions are used only for local input mapping. GNOME Portal CONTROL no longer waits for a consent dialog or the first video frame. It announces the selected desktop before reporting secure-desktop state and receiving the official viewer's “auto unlock host” request. The official Android viewer requires this desktop selection before entering its auto-unlock flow, so the notification must precede capture readiness; actual screen dimensions, codec information and capture permissions still follow their real readiness state. Enable auto unlock in the viewer and use the current Linux user's login password. The host verifies it with PAM, unlocks the existing GNOME session, requests display wake, and restores capture, input and clipboard through the shared Portal session. Initial consent or revoked permissions still require local approval. Reconnect if physical or logical dimensions change. diff --git a/README.md b/README.md index 06ef0a2..070e1f6 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ GNOME Wayland 的 Portal/EIS 路径已在 Ubuntu 26.04、GNOME 50、单屏 4K/20 Wayland 下的限制：只支持单屏、固定分辨率；Hyprland 原生后端默认使用双缓冲流水线 shm screencopy，控制端帧率设置会同步到采集节拍和编码器，但实际帧率仍受合成器、渲染和读回速度限制（旧串行路径在 2560x1600 曾测得约 16～22 帧；流水线只在虚拟机里测量过，不代表实机保证）；portal 后端暂不支持动态分辨率；GNOME 的屏幕唤醒与锁屏恢复见下；剪贴板只在控制端的远控窗口处于前台时同步。 -GNOME 缩放桌面在使用 Portal 原生键鼠时，若支持 `ConnectToEIS` 和屏幕 `mapping_id`，通过同一授权会话的 EIS 接口注入输入，按匹配的区域与实际帧尺寸分别映射横纵坐标，没有固定分辨率或缩放倍率，可避免 `NotifyPointerMotionAbsolute` 在缩放时拒绝坐标。Portal 构建需要 libei ≥ 1.1。缺少 EIS 支持时保留旧输入路径及其既有缩放限制；其他初始化错误会明确报告并触发会话恢复。EIS 连接成功后全部输入统一使用 EIS。只采集画面或使用 uinput 时不会初始化 EIS；缺少可选的逻辑尺寸也不阻断采集。键盘沿用项目现有的 Windows VK → evdev 映射。恢复时保持编码器的采集帧尺寸不变，允许缩放后的逻辑尺寸变化。已开始推流的 Portal 采集若暂停且无有效协商格式持续五秒，将进入会话恢复；静态桌面不会触发该超时。Ubuntu 26.04、GNOME Wayland 单屏 3840×2160 已实测保持连接切换 200%→166.67%→200%，恢复后画面持续更新且鼠标可控制全屏；切换期间会短暂停顿、黑屏后恢复，五秒是故障判定阈值，并非总恢复时间。此路径仍需实机验证锁屏恢复；多屏、连接中改变分辨率及旋转/镜像变换不属于本次修复范围。 捕获设置协议仍为版本 4：对客户端上报的分辨率统一使用像素坐标、DPR 为 1；Portal 逻辑尺寸仅用于本地输入映射。 +GNOME 缩放桌面在使用 Portal 原生键鼠时，若支持 `ConnectToEIS` 和屏幕 `mapping_id`，通过同一授权会话的 EIS 接口注入输入，按匹配的区域与实际帧尺寸分别映射横纵坐标，没有固定分辨率或缩放倍率，可避免 `NotifyPointerMotionAbsolute` 在缩放时拒绝坐标。Portal 构建需要 libei ≥ 1.1。缺少 EIS 支持时保留旧输入路径及其既有缩放限制；其他初始化错误会明确报告并触发会话恢复。EIS 连接成功后全部输入统一使用 EIS。只采集画面或使用 uinput 时不会初始化 EIS；缺少可选的逻辑尺寸也不阻断采集。键盘沿用项目现有的 Windows VK → evdev 映射。恢复时保持编码器的采集帧尺寸不变，允许缩放后的逻辑尺寸变化。GNOME 显示配置变化后，已推流的 Portal 采集若持续「暂停且无有效格式」，会在关联通知后的 200 毫秒宽限结束后提前进入现有恢复流程；正常重协商恢复会取消提前判定，连续通知不会推迟同次异常的判定。未收到相关通知时保留五秒兜底；静态画面或有效格式下的暂停不会触发恢复。Ubuntu 26.04、GNOME Wayland 单屏 3840×2160 已实测保持连接切换 200%→166.67%→200%，恢复后画面持续更新且鼠标可控制全屏；切换期间会短暂停顿、黑屏后恢复，五秒是兜底判定阈值，并非总恢复时间。提前恢复的测量结果和验证范围见 [Issue #27 实验记录](docs/experiments/issue-27-display-recovery.md)。此路径仍需实机验证锁屏恢复；多屏、连接中改变分辨率及旋转/镜像变换不属于本次修复范围。 捕获设置协议仍为版本 4：对客户端上报的分辨率统一使用像素坐标、DPR 为 1；Portal 逻辑尺寸仅用于本地输入映射。 GNOME Portal 的控制通道不再等待授权窗口或第一帧画面：先通知客户端当前选择桌面，再上报锁屏状态、接收官方客户端的「自动解锁被控端」请求。官方 Android 客户端需要桌面目标通知才会进入自动解锁流程，因此该通知不能等待采集就绪；真实屏幕尺寸、编码信息及采集权限仍按实际就绪状态上报。请在客户端启用自动解锁，使用当前 Linux 用户的登录密码；主机通过 PAM 验证后解锁当前 GNOME 会话、请求点亮屏幕，并恢复同一 Portal 会话中的画面、键鼠和剪贴板。首次授权或权限被撤销时仍需本机确认。物理或逻辑分辨率变化后需要重连。 diff --git a/c_engine/portal_engine.c b/c_engine/portal_engine.c index 24e660b..7e46053 100644 --- a/c_engine/portal_engine.c +++ b/c_engine/portal_engine.c @@ -24,6 +24,9 @@ #define REQUEST "org.freedesktop.portal.Request" #define SESSION "org.freedesktop.portal.Session" #define PORTAL_RENEGOTIATION_TIMEOUT (5 * G_TIME_SPAN_SECOND) +#define PORTAL_DISPLAY_GRACE (200 * G_TIME_SPAN_MILLISECOND) +#define MUTTER_DISPLAY "org.gnome.Mutter.DisplayConfig" +#define MUTTER_DISPLAY_PATH "/org/gnome/Mutter/DisplayConfig" struct uu_portal { GDBusConnection *bus; @@ -33,7 +36,7 @@ struct uu_portal { int dbus_started; char *session; char *restore_token; - guint closed_subscription, clipboard_owner_subscription, clipboard_transfer_subscription; + guint closed_subscription, clipboard_owner_subscription, clipboard_transfer_subscription, display_subscription; int clipboard_enabled, clipboard_changed; char *clipboard_text, *clipboard_mime; size_t clipboard_length; @@ -51,7 +54,7 @@ struct uu_portal { int error; enum pw_stream_state state; int format_valid, stream_started; - gint64 interrupted_at; + gint64 interrupted_at, display_changed_at; uint32_t node, width, height; int logical_width, logical_height; int absolute_stream_pixels; @@ -176,14 +179,31 @@ static int has_interface(struct uu_portal *p, const char *interface) { static void update_interruption(struct uu_portal *p) { if (p->stream_started && p->state == PW_STREAM_STATE_PAUSED && !p->format_valid) { if (!p->interrupted_at) p->interrupted_at = g_get_monotonic_time(); - } else p->interrupted_at = 0; + } else { + if (p->interrupted_at) p->display_changed_at = 0; + p->interrupted_at = 0; + } +} +static void monitors_changed(GDBusConnection *bus, const gchar *sender, const gchar *path, + const gchar *interface, const gchar *signal, GVariant *parameters, gpointer data) { + struct uu_portal *p = data; + pthread_mutex_lock(&p->lock); + /* #27: measured NULL format 10-36ms after the notification, with PAUSED + * 5-9ms before it. Preserve a correlated signal across repeated changes. */ + if (p->stream_started && (!p->display_changed_at || !p->interrupted_at || + p->display_changed_at < p->interrupted_at - PORTAL_DISPLAY_GRACE)) + p->display_changed_at = g_get_monotonic_time(); + pthread_mutex_unlock(&p->lock); } static void stream_state(void *data, enum pw_stream_state old, enum pw_stream_state state, const char *message) { (void)old; (void)message; struct uu_portal *p = data; pthread_mutex_lock(&p->lock); p->state = state; - if (state == PW_STREAM_STATE_STREAMING) p->stream_started = 1; + if (state == PW_STREAM_STATE_STREAMING) { + p->stream_started = 1; + p->display_changed_at = 0; + } update_interruption(p); pthread_mutex_unlock(&p->lock); if (state == PW_STREAM_STATE_ERROR || state == PW_STREAM_STATE_UNCONNECTED) fail(data, 9); @@ -213,6 +233,7 @@ static void stream_format(void *data, uint32_t id, const struct spa_pod *param) p->width = info.size.width; p->height = info.size.height; p->format = info.format; p->frame_length = length; p->format_valid = 1; + p->display_changed_at = 0; update_interruption(p); pthread_cond_broadcast(&p->changed); pthread_mutex_unlock(&p->lock); @@ -444,6 +465,9 @@ static struct uu_portal *open_persist(const char *restore, void (*save_token)(co g_variant_unref(results); if (*error) goto done; p->main_loop = g_main_loop_new(p->context, FALSE); + if (p->absolute_stream_pixels) + p->display_subscription = g_dbus_connection_signal_subscribe(p->bus, MUTTER_DISPLAY, MUTTER_DISPLAY, + "MonitorsChanged", MUTTER_DISPLAY_PATH, NULL, G_DBUS_SIGNAL_FLAGS_NONE, monitors_changed, p, NULL); g_main_context_pop_thread_default(p->context); if (pthread_create(&p->dbus_thread, NULL, run_dbus, p)) { *error = 3; p->cancelled = NULL; p->cancel_data = NULL; @@ -496,6 +520,7 @@ void uu_portal_close(struct uu_portal *p) { if (p->clipboard_owner_subscription) g_dbus_connection_signal_unsubscribe(p->bus,p->clipboard_owner_subscription); if (p->clipboard_transfer_subscription) g_dbus_connection_signal_unsubscribe(p->bus,p->clipboard_transfer_subscription); if (p->closed_subscription) g_dbus_connection_signal_unsubscribe(p->bus, p->closed_subscription); + if (p->display_subscription) g_dbus_connection_signal_unsubscribe(p->bus, p->display_subscription); if (p->dbus_started) { /* A queued source also handles close racing with thread startup; an * immediate g_main_loop_quit before run would leave run blocking. */ @@ -503,6 +528,8 @@ void uu_portal_close(struct uu_portal *p) { g_source_set_callback(quit, quit_dbus_loop, p->main_loop, NULL); g_source_attach(quit, p->context); g_source_unref(quit); g_main_context_wakeup(p->context); pthread_join(p->dbus_thread, NULL); + } else if (p->context) { + while (g_main_context_iteration(p->context, FALSE)) {} } if (p->pw_loop) pw_thread_loop_stop(p->pw_loop); if (p->stream) pw_stream_destroy(p->stream); @@ -531,8 +558,11 @@ int uu_portal_capture(struct uu_portal *p, uint8_t *frame, size_t length) { * paused, format-less renegotiation observed after a live scaling change (#25). * Called with the frame mutex held; recovery runs on the Rust capture worker. */ static int stream_stalled(struct uu_portal *p) { - return p->state == PW_STREAM_STATE_PAUSED && !p->format_valid && p->interrupted_at && - g_get_monotonic_time() - p->interrupted_at >= PORTAL_RENEGOTIATION_TIMEOUT; + if (p->state != PW_STREAM_STATE_PAUSED || p->format_valid || !p->interrupted_at) return 0; + gint64 now = g_get_monotonic_time(); + return now - p->interrupted_at >= PORTAL_RENEGOTIATION_TIMEOUT || + (p->display_changed_at && p->display_changed_at >= p->interrupted_at - PORTAL_DISPLAY_GRACE && + now - MAX(p->interrupted_at, p->display_changed_at) >= PORTAL_DISPLAY_GRACE); } int uu_portal_capture_since(struct uu_portal *p, uint8_t *frame, size_t length, uint64_t *revision) { if (!p || !frame) return 11; diff --git a/docs/experiments/issue-27-display-recovery.md b/docs/experiments/issue-27-display-recovery.md new file mode 100644 index 0000000..43fd88d --- /dev/null +++ b/docs/experiments/issue-27-display-recovery.md @@ -0,0 +1,82 @@ +# Issue 27: display-change-assisted Portal recovery + +## Policy + +GNOME Portal handles subscribe to Mutter's `MonitorsChanged` on their existing +private D-Bus context. The callback records a monotonic timestamp under the frame +mutex; it does not rebuild a session or block the D-Bus loop. + +A previously streaming capture can report unhealthy early only while continuously +`PAUSED` with no valid negotiated format. A notification received within 200 ms +before this interruption, or during it, enables a 200 ms grace from the later of +notification receipt and interruption start. Repeated notifications during the +same invalid pause do not extend that deadline. Accepted valid format or return +to `STREAMING` discards the notification hint. + +The existing five-second timeout remains independent. Missing Mutter signals, +non-GNOME desktops and unrelated interruptions keep that fallback. Static video, +a valid-format pause and display notifications alone do not trigger replacement. +Capture failure does not mark input or clipboard unhealthy. Existing Rust recovery +still serializes replacement, preserves permission handling and backs off errors; +its one-second lock-state grace remains unchanged. + +The 200 ms value is an initial engineering choice based on measured event skew, +not a demonstrated upper bound on all healthy renegotiations. The grace allows +natural recovery and the existing Rust health recheck cancels replacement if the +stream becomes healthy before the opener runs. + +## Measurements + +Environment: Ubuntu 26.04, GNOME 50 Wayland, one 3840x2160 display. Tests changed +only scale between 200% and 166.67%; physical frame dimensions stayed unchanged. + +Before implementation, direct numeric callback probes used the same monotonic +clock as the D-Bus observer. Across four interruptions, `PAUSED` preceded display +notification receipt by 5.3-8.6 ms; first NULL format followed it by 9.8-36.3 ms. +Three single transitions and a two-second-spaced pair recovered through the +original timeout. Notification-to-session-restored was about 6.15-6.18 s; +post-replacement capture read was about 6.17-6.21 s. A user estimated roughly +three seconds frozen plus two seconds black in one official-viewer trial; those +visual estimates were not synchronized to host timestamps. + +The candidate ran as an independent real Portal capture/H.264 encoder probe on +the same desktop, alongside the unchanged production host. It did not register +another cloud host or save screen contents. Results: + +| Trial | Notification to restored session | +| --- | ---: | +| 166.67% to 200% | 1.396 s | +| 200% to 166.67% | 1.360 s | +| Rapid pair, first notification to restore | 1.352 s | + +The rapid pair's notifications were 163 ms apart and produced one replacement. +A subsequent 30-second static observation produced no additional replacement. +The probe completed 3,600 encoded frames with an initial IDR. Display scale was +restored to its starting 166.67%; temporary Mutter changes did not overwrite the +saved display configuration. + +Candidate timing uses monotonic notification receipt and observation of the +restored log in a file polled every 5 ms. It indicates approximately 4.8 seconds +less host recovery delay on this machine. It does not measure official-viewer +rendering, pointer acceptance or black-screen duration for the candidate. No +successful real natural-renegotiation sample was observed; cancellation is covered +by deterministic tests. Other GNOME versions remain unverified. Multi-monitor, +physical-resolution and rotation support are outside this change. + +## Validation + +- `cargo test --locked`: 183 passed, 5 ignored. +- `cargo test --locked --features wayland,portal`: 191 passed, 6 ignored. +- `cargo build --locked --release --features wayland,portal`: passed. +- Native Portal engine, input, EIS and desktop-power suites passed ASan/UBSan; + EIS integration covered 13 geometries. +- Deterministic cases cover all six event orders, grace boundaries, repeated and + delayed notifications, stale hints, startup, static video, valid pauses, + natural recovery before/after the deadline, and clipboard health during an + accelerated capture timeout. +- A private test bus verifies actual notification delivery and queued-signal + teardown both with a D-Bus worker and before worker startup. + +The installed/running production binary and service configuration were unchanged +during this validation. Official-viewer acceptance of the candidate remains a +separate final check, not evidence supplied by the independent encoder probe. diff --git a/tests/portal_engine.c b/tests/portal_engine.c index 1a05491..ea3a1ec 100644 --- a/tests/portal_engine.c +++ b/tests/portal_engine.c @@ -184,6 +184,148 @@ static void portal_changed_physical_format_still_fails(void) { assert(uu_portal_capture_since(p, pixels, sizeof(pixels), NULL) == 8); uu_portal_close(p); } +static void display_change(struct uu_portal *p) { + monitors_changed(NULL, NULL, NULL, NULL, NULL, NULL, p); +} +static struct uu_portal *streaming_portal(void) { + struct uu_portal *p = test_portal(); + test_format(p, 2, 2); + stream_state(p, PW_STREAM_STATE_UNCONNECTED, PW_STREAM_STATE_STREAMING, NULL); + test_frame(p); + return p; +} +static void portal_display_change_recovery(void) { + /* Notification, PAUSED and NULL can arrive in any order. */ + const unsigned orders[][3] = {{0,1,2}, {0,2,1}, {1,0,2}, {1,2,0}, {2,0,1}, {2,1,0}}; + for (size_t order = 0; order < G_N_ELEMENTS(orders); order++) { + test_monotonic_time = 10000000; + struct uu_portal *p = streaming_portal(); + for (unsigned n = 0; n < 3; n++) { + test_monotonic_time += 20 * G_TIME_SPAN_MILLISECOND; + switch (orders[order][n]) { + case 0: display_change(p); break; + case 1: stream_state(p, PW_STREAM_STATE_STREAMING, PW_STREAM_STATE_PAUSED, NULL); break; + case 2: stream_format(p, SPA_PARAM_Format, NULL); break; + } + } + gint64 deadline = MAX(p->display_changed_at, p->interrupted_at) + PORTAL_DISPLAY_GRACE; + test_monotonic_time = deadline - 1; + assert(uu_portal_capture_alive(p)); + test_monotonic_time++; + assert(!uu_portal_capture_alive(p) && uu_portal_alive(p)); + p->clipboard_enabled = 1; + assert(uu_portal_clipboard_ready(p)); + uint8_t pixels[16]; + assert(uu_portal_capture(p, pixels, sizeof(pixels)) == 9); + uu_portal_close(p); + } + /* A delayed notification still accelerates an existing invalid pause. */ + test_monotonic_time = 20000000; + struct uu_portal *p = streaming_portal(); + stream_state(p, PW_STREAM_STATE_STREAMING, PW_STREAM_STATE_PAUSED, NULL); + stream_format(p, SPA_PARAM_Format, NULL); + test_monotonic_time += G_TIME_SPAN_SECOND; + display_change(p); + gint64 notified_at = p->display_changed_at; + test_monotonic_time += 100 * G_TIME_SPAN_MILLISECOND; + display_change(p); + assert(p->display_changed_at == notified_at); /* Repeated signals cannot postpone recovery. */ + test_monotonic_time = notified_at + PORTAL_DISPLAY_GRACE; + assert(!uu_portal_capture_alive(p) && uu_portal_alive(p)); + uu_portal_close(p); +} +static void portal_display_change_healthy_and_stale(void) { + test_monotonic_time = 30000000; + struct uu_portal *p = test_portal(); + display_change(p); + assert(!p->display_changed_at); /* Ignore notifications during startup. */ + uu_portal_close(p); + + p = streaming_portal(); + display_change(p); + test_monotonic_time += 60 * G_TIME_SPAN_SECOND; + assert(uu_portal_capture_alive(p)); /* No new frames on a static screen is healthy. */ + stream_state(p, PW_STREAM_STATE_STREAMING, PW_STREAM_STATE_PAUSED, NULL); + assert(uu_portal_capture_alive(p)); /* A valid format must not trigger recovery. */ + stream_format(p, SPA_PARAM_Format, NULL); + gint64 invalid_at = test_monotonic_time; + test_monotonic_time += PORTAL_DISPLAY_GRACE; + assert(uu_portal_capture_alive(p)); /* The notification from a minute ago is unrelated. */ + test_monotonic_time = invalid_at + PORTAL_RENEGOTIATION_TIMEOUT; + assert(!uu_portal_capture_alive(p)); + uu_portal_close(p); + + test_monotonic_time = 100000000; + p = streaming_portal(); + display_change(p); + stream_state(p, PW_STREAM_STATE_STREAMING, PW_STREAM_STATE_PAUSED, NULL); + stream_format(p, SPA_PARAM_Format, NULL); + test_monotonic_time += PORTAL_DISPLAY_GRACE - 1; + test_format(p, 2, 2); /* Natural renegotiation wins before the early check. */ + assert(!p->display_changed_at && !p->interrupted_at); + test_monotonic_time += G_TIME_SPAN_SECOND; + assert(uu_portal_capture_alive(p)); + stream_format(p, SPA_PARAM_Format, NULL); + test_monotonic_time += PORTAL_DISPLAY_GRACE; + assert(uu_portal_capture_alive(p)); /* Recovered signal cannot accelerate another failure. */ + display_change(p); /* A later display change correlates with the retained invalid state. */ + test_monotonic_time += PORTAL_DISPLAY_GRACE; + assert(!uu_portal_capture_alive(p) && uu_portal_alive(p)); + /* Even after the fast deadline, recovery can still win before replacement. */ + test_format(p, 2, 2); + assert(uu_portal_capture_alive(p) && !p->display_changed_at); + display_change(p); + test_format(p, 2, 2); /* Successful format without an invalid pause discards its hint. */ + assert(!p->display_changed_at); + display_change(p); + stream_state(p, PW_STREAM_STATE_PAUSED, PW_STREAM_STATE_STREAMING, NULL); + assert(!p->display_changed_at); + uu_portal_close(p); +} +static void portal_display_subscription_lifecycle(void) { + GTestDBus *test_bus = g_test_dbus_new(G_TEST_DBUS_NONE); + g_test_dbus_up(test_bus); + GDBusConnection *bus = g_dbus_connection_new_for_address_sync(g_test_dbus_get_bus_address(test_bus), + G_DBUS_CONNECTION_FLAGS_AUTHENTICATION_CLIENT | G_DBUS_CONNECTION_FLAGS_MESSAGE_BUS_CONNECTION, + NULL, NULL, NULL); + assert(bus); + const char *sender = g_dbus_connection_get_unique_name(bus); + for (unsigned threaded = 0; threaded < 2; threaded++) { + struct uu_portal *p = streaming_portal(); + p->bus = g_object_ref(bus); + p->context = g_main_context_new(); + p->main_loop = g_main_loop_new(p->context, FALSE); + g_main_context_push_thread_default(p->context); + p->display_subscription = g_dbus_connection_signal_subscribe(bus, sender, MUTTER_DISPLAY, + "MonitorsChanged", MUTTER_DISPLAY_PATH, NULL, G_DBUS_SIGNAL_FLAGS_NONE, monitors_changed, p, NULL); + g_main_context_pop_thread_default(p->context); + /* A synchronous roundtrip places the match rule before the signal. */ + GVariant *reply = g_dbus_connection_call_sync(bus, "org.freedesktop.DBus", "/org/freedesktop/DBus", + "org.freedesktop.DBus", "GetId", NULL, NULL, G_DBUS_CALL_FLAGS_NONE, 3000, NULL, NULL); + assert(reply); g_variant_unref(reply); + assert(g_dbus_connection_emit_signal(bus, NULL, MUTTER_DISPLAY_PATH, MUTTER_DISPLAY, + "MonitorsChanged", g_variant_new("()"), NULL)); + assert(g_dbus_connection_flush_sync(bus, NULL, NULL)); + /* The test clock is fixed; use a bounded iteration count for delivery. */ + for (unsigned attempt = 0; attempt < 1000 && !p->display_changed_at; attempt++) { + while (g_main_context_iteration(p->context, FALSE)) {} + if (!p->display_changed_at) g_usleep(1000); + } + assert(p->display_changed_at); + if (threaded) { + assert(!pthread_create(&p->dbus_thread, NULL, run_dbus, p)); + p->dbus_started = 1; + } + assert(g_dbus_connection_emit_signal(bus, NULL, MUTTER_DISPLAY_PATH, MUTTER_DISPLAY, + "MonitorsChanged", g_variant_new("()"), NULL)); + assert(g_dbus_connection_flush_sync(bus, NULL, NULL)); + uu_portal_close(p); /* Queued signals and immediate loop shutdown must be safe. */ + } + g_dbus_connection_close_sync(bus, NULL, NULL); + g_object_unref(bus); + g_test_dbus_down(test_bus); + g_object_unref(test_bus); +} static int cancel_immediately(void *data) { return *(const int *)data; } struct cancel_later { int *cancelled; int polls; }; @@ -282,6 +424,9 @@ int main(void) { portal_renegotiation_recovery(); portal_renegotiation_event_order_and_healthy_states(); portal_changed_physical_format_still_fails(); + portal_display_change_recovery(); + portal_display_change_healthy_and_stale(); + portal_display_subscription_lifecycle(); cancellable_waits(); clipboard_bounds(); capture_with_optional_geometry(0); From 517c4aee96e40a35f6489dd94017f829108b456d Mon Sep 17 00:00:00 2001 From: VallierDev <24893073+VallierDev@users.noreply.github.com> Date: Sat, 17 Oct 2026 14:24:08 +0800 Subject: [PATCH 2/2] docs: keep the #27 experiment record in the PR instead of the repository Co-Authored-By: Claude Opus 5.5 --- README.en.md | 2 +- README.md | 2 +- docs/experiments/issue-27-display-recovery.md | 82 ------------------- 3 files changed, 2 insertions(+), 84 deletions(-) delete mode 100644 docs/experiments/issue-27-display-recovery.md diff --git a/README.en.md b/README.en.md index e8f1eb8..310d379 100644 --- a/README.en.md +++ b/README.en.md @@ -74,7 +74,7 @@ real hardware, please [open an issue](../../issues) with your distribution, desk Wayland limits: single screen, fixed resolution. The native Hyprland backend now pipelines two shm screencopy buffers and passes the viewer's requested fps to both capture pacing and the encoder. Actual fps remains limited by the compositor, rendering and readback (the old serial path measured about 16–22 fps at 2560x1600; the pipeline is VM-measured only, not a hardware guarantee). The portal backend does not support dynamic resolution; see below for GNOME wake and lock recovery; the clipboard syncs while the viewer's remote window has focus. On the first connection the desktop asks for "Remote Desktop / Screen Sharing" permission; the permission is remembered once granted. -Scaled GNOME desktops using native Portal input use EIS on the same authorized session when `ConnectToEIS` and a screen `mapping_id` are available. Each axis is mapped using the matched region and actual frame dimensions, with no fixed resolution or scale factor. This avoids scaled-coordinate rejection by `NotifyPointerMotionAbsolute`. Portal builds require libei >= 1.1. Missing EIS support retains the legacy input path and its existing scaling limitations; other initialization failures are reported and trigger session recovery. After a successful EIS connection, all input uses EIS. Capture alone and uinput do not initialize EIS; missing optional logical dimensions do not block capture. Keyboard input uses the existing Windows VK to evdev mapping. Recovery keeps the encoder frame dimensions fixed but accepts logical-size changes after scaling. On GNOME, a display-change notification enables earlier recovery when previously streaming Portal capture remains paused without a valid format for a 200 ms grace after both conditions are present. Healthy renegotiation cancels the early check; repeated notifications do not postpone the same interruption. The five-second fallback remains without a related notification. Static video and valid-format pauses do not trigger recovery. Tested while connected on Ubuntu 26.04, GNOME Wayland, one 3840x2160 display: 200% -> 166.67% -> 200%, with continuously updating video and full-screen pointer control after recovery. Switching briefly freezes and goes black before recovering; five seconds is the fallback threshold, not total recovery time. See the [Issue #27 experiment](docs/experiments/issue-27-display-recovery.md) for timing measurements and validation limits. Lock recovery still needs hardware acceptance testing; multi-monitor support, live resolution changes, and rotation/reflection transforms are outside this fix's scope. Capture settings remain at protocol version 4: advertised resolutions use pixel coordinates and DPR 1; Portal logical dimensions are used only for local input mapping. +Scaled GNOME desktops using native Portal input use EIS on the same authorized session when `ConnectToEIS` and a screen `mapping_id` are available. Each axis is mapped using the matched region and actual frame dimensions, with no fixed resolution or scale factor. This avoids scaled-coordinate rejection by `NotifyPointerMotionAbsolute`. Portal builds require libei >= 1.1. Missing EIS support retains the legacy input path and its existing scaling limitations; other initialization failures are reported and trigger session recovery. After a successful EIS connection, all input uses EIS. Capture alone and uinput do not initialize EIS; missing optional logical dimensions do not block capture. Keyboard input uses the existing Windows VK to evdev mapping. Recovery keeps the encoder frame dimensions fixed but accepts logical-size changes after scaling. On GNOME, a display-change notification enables earlier recovery when previously streaming Portal capture remains paused without a valid format for a 200 ms grace after both conditions are present. Healthy renegotiation cancels the early check; repeated notifications do not postpone the same interruption. The five-second fallback remains without a related notification. Static video and valid-format pauses do not trigger recovery. Tested while connected on Ubuntu 26.04, GNOME Wayland, one 3840x2160 display: 200% -> 166.67% -> 200%, with continuously updating video and full-screen pointer control after recovery. Switching briefly freezes and goes black before recovering; five seconds is the fallback threshold, not total recovery time. See [#35](../../pull/35) for timing measurements and validation limits. Lock recovery still needs hardware acceptance testing; multi-monitor support, live resolution changes, and rotation/reflection transforms are outside this fix's scope. Capture settings remain at protocol version 4: advertised resolutions use pixel coordinates and DPR 1; Portal logical dimensions are used only for local input mapping. GNOME Portal CONTROL no longer waits for a consent dialog or the first video frame. It announces the selected desktop before reporting secure-desktop state and receiving the official viewer's “auto unlock host” request. The official Android viewer requires this desktop selection before entering its auto-unlock flow, so the notification must precede capture readiness; actual screen dimensions, codec information and capture permissions still follow their real readiness state. Enable auto unlock in the viewer and use the current Linux user's login password. The host verifies it with PAM, unlocks the existing GNOME session, requests display wake, and restores capture, input and clipboard through the shared Portal session. Initial consent or revoked permissions still require local approval. Reconnect if physical or logical dimensions change. diff --git a/README.md b/README.md index 070e1f6..fa3f3d0 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ GNOME Wayland 的 Portal/EIS 路径已在 Ubuntu 26.04、GNOME 50、单屏 4K/20 Wayland 下的限制：只支持单屏、固定分辨率；Hyprland 原生后端默认使用双缓冲流水线 shm screencopy，控制端帧率设置会同步到采集节拍和编码器，但实际帧率仍受合成器、渲染和读回速度限制（旧串行路径在 2560x1600 曾测得约 16～22 帧；流水线只在虚拟机里测量过，不代表实机保证）；portal 后端暂不支持动态分辨率；GNOME 的屏幕唤醒与锁屏恢复见下；剪贴板只在控制端的远控窗口处于前台时同步。 -GNOME 缩放桌面在使用 Portal 原生键鼠时，若支持 `ConnectToEIS` 和屏幕 `mapping_id`，通过同一授权会话的 EIS 接口注入输入，按匹配的区域与实际帧尺寸分别映射横纵坐标，没有固定分辨率或缩放倍率，可避免 `NotifyPointerMotionAbsolute` 在缩放时拒绝坐标。Portal 构建需要 libei ≥ 1.1。缺少 EIS 支持时保留旧输入路径及其既有缩放限制；其他初始化错误会明确报告并触发会话恢复。EIS 连接成功后全部输入统一使用 EIS。只采集画面或使用 uinput 时不会初始化 EIS；缺少可选的逻辑尺寸也不阻断采集。键盘沿用项目现有的 Windows VK → evdev 映射。恢复时保持编码器的采集帧尺寸不变，允许缩放后的逻辑尺寸变化。GNOME 显示配置变化后，已推流的 Portal 采集若持续「暂停且无有效格式」，会在关联通知后的 200 毫秒宽限结束后提前进入现有恢复流程；正常重协商恢复会取消提前判定，连续通知不会推迟同次异常的判定。未收到相关通知时保留五秒兜底；静态画面或有效格式下的暂停不会触发恢复。Ubuntu 26.04、GNOME Wayland 单屏 3840×2160 已实测保持连接切换 200%→166.67%→200%，恢复后画面持续更新且鼠标可控制全屏；切换期间会短暂停顿、黑屏后恢复，五秒是兜底判定阈值，并非总恢复时间。提前恢复的测量结果和验证范围见 [Issue #27 实验记录](docs/experiments/issue-27-display-recovery.md)。此路径仍需实机验证锁屏恢复；多屏、连接中改变分辨率及旋转/镜像变换不属于本次修复范围。 捕获设置协议仍为版本 4：对客户端上报的分辨率统一使用像素坐标、DPR 为 1；Portal 逻辑尺寸仅用于本地输入映射。 +GNOME 缩放桌面在使用 Portal 原生键鼠时，若支持 `ConnectToEIS` 和屏幕 `mapping_id`，通过同一授权会话的 EIS 接口注入输入，按匹配的区域与实际帧尺寸分别映射横纵坐标，没有固定分辨率或缩放倍率，可避免 `NotifyPointerMotionAbsolute` 在缩放时拒绝坐标。Portal 构建需要 libei ≥ 1.1。缺少 EIS 支持时保留旧输入路径及其既有缩放限制；其他初始化错误会明确报告并触发会话恢复。EIS 连接成功后全部输入统一使用 EIS。只采集画面或使用 uinput 时不会初始化 EIS；缺少可选的逻辑尺寸也不阻断采集。键盘沿用项目现有的 Windows VK → evdev 映射。恢复时保持编码器的采集帧尺寸不变，允许缩放后的逻辑尺寸变化。GNOME 显示配置变化后，已推流的 Portal 采集若持续「暂停且无有效格式」，会在关联通知后的 200 毫秒宽限结束后提前进入现有恢复流程；正常重协商恢复会取消提前判定，连续通知不会推迟同次异常的判定。未收到相关通知时保留五秒兜底；静态画面或有效格式下的暂停不会触发恢复。Ubuntu 26.04、GNOME Wayland 单屏 3840×2160 已实测保持连接切换 200%→166.67%→200%，恢复后画面持续更新且鼠标可控制全屏；切换期间会短暂停顿、黑屏后恢复，五秒是兜底判定阈值，并非总恢复时间。提前恢复的测量结果和验证范围见 [#35](../../pull/35)。此路径仍需实机验证锁屏恢复；多屏、连接中改变分辨率及旋转/镜像变换不属于本次修复范围。 捕获设置协议仍为版本 4：对客户端上报的分辨率统一使用像素坐标、DPR 为 1；Portal 逻辑尺寸仅用于本地输入映射。 GNOME Portal 的控制通道不再等待授权窗口或第一帧画面：先通知客户端当前选择桌面，再上报锁屏状态、接收官方客户端的「自动解锁被控端」请求。官方 Android 客户端需要桌面目标通知才会进入自动解锁流程，因此该通知不能等待采集就绪；真实屏幕尺寸、编码信息及采集权限仍按实际就绪状态上报。请在客户端启用自动解锁，使用当前 Linux 用户的登录密码；主机通过 PAM 验证后解锁当前 GNOME 会话、请求点亮屏幕，并恢复同一 Portal 会话中的画面、键鼠和剪贴板。首次授权或权限被撤销时仍需本机确认。物理或逻辑分辨率变化后需要重连。 diff --git a/docs/experiments/issue-27-display-recovery.md b/docs/experiments/issue-27-display-recovery.md deleted file mode 100644 index 43fd88d..0000000 --- a/docs/experiments/issue-27-display-recovery.md +++ /dev/null @@ -1,82 +0,0 @@ -# Issue 27: display-change-assisted Portal recovery - -## Policy - -GNOME Portal handles subscribe to Mutter's `MonitorsChanged` on their existing -private D-Bus context. The callback records a monotonic timestamp under the frame -mutex; it does not rebuild a session or block the D-Bus loop. - -A previously streaming capture can report unhealthy early only while continuously -`PAUSED` with no valid negotiated format. A notification received within 200 ms -before this interruption, or during it, enables a 200 ms grace from the later of -notification receipt and interruption start. Repeated notifications during the -same invalid pause do not extend that deadline. Accepted valid format or return -to `STREAMING` discards the notification hint. - -The existing five-second timeout remains independent. Missing Mutter signals, -non-GNOME desktops and unrelated interruptions keep that fallback. Static video, -a valid-format pause and display notifications alone do not trigger replacement. -Capture failure does not mark input or clipboard unhealthy. Existing Rust recovery -still serializes replacement, preserves permission handling and backs off errors; -its one-second lock-state grace remains unchanged. - -The 200 ms value is an initial engineering choice based on measured event skew, -not a demonstrated upper bound on all healthy renegotiations. The grace allows -natural recovery and the existing Rust health recheck cancels replacement if the -stream becomes healthy before the opener runs. - -## Measurements - -Environment: Ubuntu 26.04, GNOME 50 Wayland, one 3840x2160 display. Tests changed -only scale between 200% and 166.67%; physical frame dimensions stayed unchanged. - -Before implementation, direct numeric callback probes used the same monotonic -clock as the D-Bus observer. Across four interruptions, `PAUSED` preceded display -notification receipt by 5.3-8.6 ms; first NULL format followed it by 9.8-36.3 ms. -Three single transitions and a two-second-spaced pair recovered through the -original timeout. Notification-to-session-restored was about 6.15-6.18 s; -post-replacement capture read was about 6.17-6.21 s. A user estimated roughly -three seconds frozen plus two seconds black in one official-viewer trial; those -visual estimates were not synchronized to host timestamps. - -The candidate ran as an independent real Portal capture/H.264 encoder probe on -the same desktop, alongside the unchanged production host. It did not register -another cloud host or save screen contents. Results: - -| Trial | Notification to restored session | -| --- | ---: | -| 166.67% to 200% | 1.396 s | -| 200% to 166.67% | 1.360 s | -| Rapid pair, first notification to restore | 1.352 s | - -The rapid pair's notifications were 163 ms apart and produced one replacement. -A subsequent 30-second static observation produced no additional replacement. -The probe completed 3,600 encoded frames with an initial IDR. Display scale was -restored to its starting 166.67%; temporary Mutter changes did not overwrite the -saved display configuration. - -Candidate timing uses monotonic notification receipt and observation of the -restored log in a file polled every 5 ms. It indicates approximately 4.8 seconds -less host recovery delay on this machine. It does not measure official-viewer -rendering, pointer acceptance or black-screen duration for the candidate. No -successful real natural-renegotiation sample was observed; cancellation is covered -by deterministic tests. Other GNOME versions remain unverified. Multi-monitor, -physical-resolution and rotation support are outside this change. - -## Validation - -- `cargo test --locked`: 183 passed, 5 ignored. -- `cargo test --locked --features wayland,portal`: 191 passed, 6 ignored. -- `cargo build --locked --release --features wayland,portal`: passed. -- Native Portal engine, input, EIS and desktop-power suites passed ASan/UBSan; - EIS integration covered 13 geometries. -- Deterministic cases cover all six event orders, grace boundaries, repeated and - delayed notifications, stale hints, startup, static video, valid pauses, - natural recovery before/after the deadline, and clipboard health during an - accelerated capture timeout. -- A private test bus verifies actual notification delivery and queued-signal - teardown both with a D-Bus worker and before worker startup. - -The installed/running production binary and service configuration were unchanged -during this validation. Official-viewer acceptance of the candidate remains a -separate final check, not evidence supplied by the independent encoder probe.