Repository navigation
spike(node): the connected device's live screen on its 3D model - #7651
Draft
jamesarich wants to merge 36 commits into
Draft
jamesarich wants to merge 36 commits into
jamesarich wants to merge 36 commits into
Conversation
Consumes the DisplayFrame PoC protobufs (2.99.0-screen-mirror-poc-SNAPSHOT from mavenLocal, useMavenLocal committed on this branch): DisplayMirrorManager reassembles FromRadio.display_frame chunks into MirrorFrame snapshots, AdminController gains setDisplayMirror + sendInputEvent, and the Debug panel gains a Mirror tab rendering the device's 1bpp framebuffer live with a D-pad driving the firmware InputBroker. PoC: tab strings deliberately unlocalized.
Cap the mirror canvas at 512.dp (4x for a 128px OLED) instead of filling the window width, fold the frame info into the toggle row, and pair Down/Back so the controls fit without scrolling on desktop.
Manager: validate format (MONO_VLSB only) and geometry (width/height must describe exactly total_size bytes) so malformed device input can't reach aspectRatio and crash; unit tests for reassembly, drops, restarts. AdminController: setDisplayMirror/sendInputEvent become non-suspend immediate sends (teardown-safe, input never queues behind bulk admin) and gain requestDisplayFrame. Mirror tab: render each frame once into a 1:1 ImageBitmap scaled with FilterQuality.None (kills the fractional-scale seams), gate controls on connection state, reset the toggle on disconnect, stop the stream on tab dispose and ViewModel clear, add a one-shot Refresh, inject AdminController/ConnectionStateProvider instead of RadioController. Dispatch: display_frame test coverage incl. revoked-session.
Populates the new DisplayInfo during the handshake — dimensions from the live display device, MONO_VLSB frame format, PanelClass from the panel defines (EINK/HUB75/TFT/LCD/OLED), has_touch from HAS_TOUCHSCREEN — so clients can gate mirroring UI and adapt to the panel. Absent on display-less builds. Regenerates protos (DisplayInfo, DisplayFrame rect fields reserved for dirty-rect streaming).
Replace the flat button stack with the researched control surface: cross-shaped 5-way cluster (56dp circular keys, 8dp gaps, Material Symbols chevrons with contentDescriptions), directions auto-repeat on hold (500ms then 10Hz — slow enough for the radio to render), OK long-press sends SELECT_LONG, Back sits below per TV-remote convention. The mirror itself becomes an input surface: click to capture the keyboard (arrows / Enter / Esc, with a visible focus ring and hint) and swipe in a cardinal direction to navigate. Consumes the new DeviceMetadata.display: shows an e-ink hint steering toward one-shot Refresh. Adds keyboard_arrow_left/right icons (Material Symbols rounded, transposed from the existing up glyph).
DisplayMirrorManager reassembles FromRadio.display_palette chunks (signature-keyed, defaults captured on the offset-0 chunk) into a MirrorPalette; frames carry palette_signature and the renderer resolves each pixel through the region table (last overlapping region wins, matching firmware precedence) with RGB565 conversion, falling back to monochrome when the referenced palette hasn't arrived. Renderer split into MirrorFrameRenderer.kt.
Controls sit beside the mirror when the content width allows (>=760dp, BoxWithConstraints) and stack below it otherwise; tapping the mirror now toggles keyboard capture instead of only acquiring it, with the hint text naming the way out.
Circular D-pad ring replaces the cross cluster (wedge hit-testing with a radial dead band, hold-to-repeat, per-wedge semantics, haptics), keyboard capture becomes an explicit chip with tap-to-toggle, and taps on touch devices forward real panel coordinates (long-press = SELECT, matching the physical touch driver). Manager gains reset() (wired to ViewModel creation and disconnect), a palette region cap, and first-chunk geometry pinning; renderer uses bit-replicated RGB565 and run-length row drawing with the pixel resolver extracted pure and tested; KeyUp of captured keys is consumed; ViewModel moved to its own file; all strings localized; the stray .bak test file is gone.
DisplayMirrorManager reassembles RGB565 rect chunks (sequence, geometry, and panel-bound validation) and composites them into a persistent little-endian RGB565 canvas emitted per completed rect; the renderer gains a true-color branch using the same run-length row drawing. Mono frames are untouched.
RGB565 frames identify LVGL/MUI devices, whose UIs read their own input drivers rather than the InputBroker remote events inject into — so the mirror hides the D-pad, keyboard, swipe, and touch affordances and says so, until an input-injection path exists. Utility helpers and the input codes move to DisplayMirrorUtils.kt.
device-ui now accepts injected input (virtual LVGL pointer + group-attached keypad bridged from the InputBroker), so the view-only treatment goes away: the D-pad navigates MUI widgets, taps forward panel coordinates, and the keyboard types. The mirror_view_only string stays for the future capability-flag path.
The mirror's keyboard capture only mapped navigation keys and silently dropped everything else, so text entry did nothing: printable characters never left the client even though the firmware already injects kb_char. Characters now go out as ANYKEY with the code point, Backspace goes as a character (so text fields delete rather than navigating back, which stays Esc's job), and the handling moves into a Modifier.remoteKeyInput extension.
Third adversarial review pass over the client side. Rect reassembly moves out of DisplayMirrorManagerImpl into MirrorRectCompositor: the manager was carrying two unrelated state machines and 14 methods. Every rect field is now validated before anything is allocated (total_size is device-controlled), geometry is latched from the chunk at offset 0 so a later chunk cannot transpose the blit, continuations must match the buffer that was actually allocated rather than their own claim, and bounds use the subtraction form because rect_x + rect_width overflows for hostile values. A rect that omits width or height covers the full panel in that axis. remoteKeyInput no longer swallows modified keys, so Cmd+Q, Ctrl+C and Alt+Tab stay with the desktop instead of being forwarded to the device. MirrorFrameRenderer tracks colorized runs as a raw Int with a sentinel rather than a boxed Color?, and MirrorFrame.hashCode includes the format so a mono and an RGB565 frame of the same size cannot collide. Drops mirror_view_only, orphaned since MUI gained remote input.
The Debug Panel is the one Settings route deliberately exempt from the `connected && !isManaged` gate every sibling carries, justified by a comment saying it "reads app-local logs only". The Mirror tab put an admin write (set_display_mirror) and arbitrary input injection on that route, so managed mode - which exists to lock a device out of configuration - was bypassed entirely. Enforced in the ViewModel, because the route will not do it: arming, one-shot frame requests, key, character and touch sends are all refused while managed. Disarming still goes through, so a stream already running is torn down when managed mode arrives mid-session. The UI gates on `active = connected && !managed` rather than offering controls that silently do nothing, and the RadioConfig comment now says what the route actually guarantees, since its old wording is what let this through. MirrorHeaderRow is split out of DisplayMirrorContent, which was at detekt's LongMethod limit and could not absorb another line.
The mirror's key capture was a focusable box, so it could never raise the soft keyboard. It now uses core/ui's RemoteKeyboardSink: the Keyboard chip raises the IME, typed text reaches the device one code point per kb_char (emoji included), Enter is OK, Esc is Back, Backspace and Tab are typed characters, and Ctrl/Alt chords stay with the host. The scroll column pads for the IME so the D-pad stays visible. Space now types a space instead of meaning OK.
…e flavor The flasher serves a real-size GLB beside each device illustration that has one. Node detail's hero avatar shows a 3D badge when the model exists and opens it full screen, orbitable and lit by a bundled CC0 studio light probe over the theme background, with the illustration as the fallback. On the google flavor an AR action places the model on a real surface at real size through ARCore; the fdroid flavor and desktop have no AR and never show it.
…frames off the main thread
…s AR and rotation
The orbit and AR viewers lay the mirror frame over the model's screen-named mesh and update it per frame. The Pocket's hardware entry points at the bare RAK4631 image, so its model is hard-coded here.
jamesarich
added
enhancement
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jamesarich
force-pushed
the
feat/device-3d-model
branch
from
October 19, 2026 23:00
4a706dc to
dd0074b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The connected device's own screen, live on its 3D model: the screen-mirror stream from #6987 drawn onto the model's display, in the orbit view and in AR.
Stacked on #7639 with #6987 merged in; only the
spike(node)commits are new. Not for merge: it carries #6987's mavenLocal protobufs pin, so CI cannot resolve dependencies.🌟 New Features
screenmesh or material. It disarms when the dialog closes, holds the stream over a rotation, and stays off in managed mode.Known gaps
rak_wismesh_pocket.glbnames its screen; the other models name parts by colour, so the model generator needs to emit ascreenpart.rak4631.svg, so this branch hard-codes its model. The real fix is in the api data.Testing Performed
GlbBoundsTestscreen cases.