[DISCUSS] Incremental painting for the GPUI backend / GPUI 后端的增量绘制

[Astro-Han]

Problem. NSH's GPUI backend draws each surface with one element, and its paint walks the whole tree (render::draw_view). Whatever changed, every frame repaints everything visible. Agents produce frames constantly: streaming text, tool progress and status changes. Moving away from Electron does not remove this cost by itself.

Measurement. Release build on a MacBook Pro built-in display: a 1280×821 window with 32 lines of text and three animated dots. Each mode ran two 20-second phases. CPU is the share of one core for the Maka process; GPU is system-wide, with other apps open.

Dots driven by	CPU	GPU
Static	1.0%	31%
Animated.loop	25.3%	69%
66 ms timer, AnimatedValue.set	5.2%	36%
125 ms timer, AnimatedValue.set	3.6%	35%

The native loop runs no JS per frame, yet it costs the most because it redraws the whole tree on every display frame. inactive_frame_interval: None keeps that rate when the window loses focus. In a fully occluded window, all modes stayed at or below 0.4%.

Not measured yet: streaming text, a long real transcript, or a window attached to many running sessions (the desktop validation still open in #6078).

Options.

1. Cached subtrees: split a surface into islands using Entity::cached, which gpui-fast already has. The hard parts are that one element owns the surface, transforms and opacity live in DrawCx, and event listeners are installed during paint.
2. Replay clean subtrees: record paint primitives per NshView and replay the unchanged ones. This is more general, but may need new gpui-fast API.
3. A frame-rate cap for looping animations: small and independent, but it does not help streaming.

Questions for @M4n5ter.

• Should this start in longbridge/nsh, your fork, or third-party/nsh?
• Which option fits the backend's design, or is there a better one?
• Can we first agree on a benchmark? I suggest one window attached to 16, 256 and 1,024 running sessions with streaming active, recording CPU, RSS and frame gaps with the existing frame statistics, then setting a budget for visual motion.

---

问题。 NSH 的 GPUI 后端用一个元素绘制整个 surface，它的 paint 会遍历整棵树（render::draw_view）。无论改了什么，每一帧都要重画所有可见内容。Agent 又会持续产生新帧：流式文字、工具进度、状态变化。离开 Electron 并不会自动消除这项成本。

测量。 Release 构建，MacBook Pro 内建屏幕，1280×821 窗口，32 行文字加三个动画点。每种模式跑两轮，每轮 20 秒。CPU 是 Maka 进程占单核的比例；GPU 是全系统占用，当时还开着其他应用。

三点驱动方式	CPU	GPU
静止	1.0%	31%
Animated.loop	25.3%	69%
66ms 定时器，AnimatedValue.set	5.2%	36%
125ms 定时器，AnimatedValue.set	3.6%	35%

原生循环每帧不运行 JS，却最贵，因为它在每个显示帧都重绘整棵树。inactive_frame_interval: None 让窗口失焦后也保持这个帧率。窗口被完全遮挡时，各模式的 CPU 都不超过 0.4%。

还没测： 流式文字、真实长对话、连着大量运行中会话的窗口（也就是 #6078 里还没做的桌面验证）。

可选方向。

1. 缓存子树： 用 gpui-fast 已有的 Entity::cached 把 surface 切成多个区块。难点是一个元素独占 surface，变换和透明度都在 DrawCx 里，事件监听也是在绘制时安装的。
2. 回放未变化的子树： 按 NshView 记录绘制图元，未变化的直接回放。这个方向更通用，但可能需要 gpui-fast 增加 API。
3. 限制循环动画的帧率： 改动小，可以独立做，但对流式输出没有帮助。

想请 @M4n5ter 一起看的问题。

• 这件事应该从 longbridge/nsh、你的 fork，还是 third-party/nsh 开始？
• 哪个方向符合后端的设计，或者有更好的做法？
• 能否先定一个基准？我建议一个窗口分别连着 16、256、1,024 个运行中的会话并开启流式输出，用现有的帧统计记录 CPU、RSS 和帧间隔，再据此定动效预算。

[M4n5ter]

English

Before adding another caching or paint-replay mechanism to the GPUI backend, I’d prefer to evaluate NSH’s existing platform-native backends for Maka desktop.

On macOS, the AppKit backend already uses per-node, layer-backed views. That looks like a more direct way to avoid the surface-wide painting path you identified. Please start with macOS: compare the same small animation case, then a real streaming conversation, using the same release profile, window size and display refresh rate. Measure process CPU, RSS and frame times before deciding the next step.

Keep the Host and shared TSX UI intact. The work should focus on desktop-shell integration and any missing native-backend embedding APIs needed to reuse our engine configuration, Host bindings and dynamic asset/module loading.

For repository ownership: longbridge/nsh lacks TUI support, which is why we originally used my fork. That fork has already been moved into third-party/nsh and is maintained there. Please work against that tree; we can periodically synchronize updates from longbridge/nsh, preserving our TUI support and local adaptations. There is no separate fork repository to keep synchronized for this work.

简体中文

在给 GPUI 后端增加另一套缓存或绘制回放机制之前，我更倾向于先评估 NSH 已有的平台原生后端是否更适合 Maka desktop。

macOS 的 AppKit 后端已经使用每个节点独立的 layer-backed view，从结构上看，更直接地避开了你指出的整个 surface 重复绘制路径。请先从 macOS 开始：用相同的 release 配置、窗口大小和显示刷新率，对比这个小动画场景，再测真实流式对话，记录进程 CPU、RSS 和帧耗时，再决定下一步。

保留现有 Host 和共享 TSX UI，改动集中在 desktop-shell 接入，以及原生后端为复用现有 engine 配置、Host bindings、动态资源和模块加载所缺少的嵌入接口上。

代码维护位置也说明一下：longbridge/nsh 缺少 TUI 支持，所以我们最初使用了我的 fork。这个 fork 早已移入 third-party/nsh，现在直接在这里维护。这项工作也请基于该目录进行，之后定期同步 longbridge/nsh，保留我们的 TUI 支持和本地适配即可，不需要另外同步维护一个 fork 仓库。

[Astro-Han]

Thanks, this matches what I wanted to check next, so I ran it. I'll work in third-party/nsh.

Result: the real workbench runs on the AppKit backend with the Host and shared TSX UI unchanged, and it costs clearly less than GPUI in every case I measured. I'd keep going with AppKit on macOS, but it isn't ready to replace GPUI yet: gesture feel, IME and a few functional gaps are still open.

Setup. The same release binary picks the backend from an environment variable. Apple M5 Pro, built-in 120 Hz display, 1280×820 window (GPUI 1280×821). The window stayed in front and any phase where it lost focus was dropped. The conversation has 200 turns of Markdown, code blocks and tables. Streaming goes through the real Host from a local OpenAI-compatible server, 30 deltas per second.

Scenario	GPUI CPU / GPU	AppKit CPU / GPU
Idle	1.2% / 5%	0.0% / 1%
Streaming a reply	15.1% / 32%	8.1% / 0%
Scrolling the list	26.6% / 37%	14.2% / 1%
Three dots, Animated.loop	29.2% / 43%	3.0% / 0%
Three dots, 66 ms timer	7.3% / 25%	1.4% / 0%
Three dots, 125 ms timer	5.0% / 19%	0.9% / 1%

CPU is the share of one core for the shell process; GPU is system-wide. Each row averages 2–6 phases of 5–15 seconds, with little spread. The dots run in a panel of 32 text lines over the workbench. Scrolling uses a 60 Hz workbench timer calling scrollToOffset at 720 px/s, with matching offsets on both backends, so it skips gesture input. I couldn't get injected wheel events to sustain scrolling.

The small probe from before, one run each: native animation 13.1% → 1.5%, 15 Hz timer 2.5% → 0.3%, 30 Hz text 6.1% → 2.3%, RSS 127 → 108 MB.

Memory	GPUI	AppKit
RSS, idle / streaming	232 / 239 MB	215 / 238 MB
footprint, idle / streaming	280 / 286 MB	112 / 128 MB

RSS barely differs. footprint does, and in GPUI's case it includes 66 MB of IOSurface and graphics memory. It doesn't count anything WindowServer holds for AppKit's layers.

Frames	GPUI	AppKit
Streaming: fps, gap p50 / p95	121, 8.3 / 9.1 ms	119, 8.3 / 8.7 ms
Scrolling: fps, gap p50 / p95	121, 8.3 / 9.2 ms	119, 8.3 / 11.1 ms

The backends count frames differently (GPUI counts paints, AppKit counts dirty refreshes), so compare these loosely. Pacing is close, but AppKit has the wider scrolling p95, and its worst main-thread frames reached about 12 ms, with up to 4% of frames in a second over one refresh. I suspect native view mounting but haven't confirmed it. That is the risk I'd watch.

WindowServer sat around 46–49% in all scenarios because of other open apps. Scrolling added 0.4 points on GPUI and 2.7 on AppKit, so I see no large shift of cost into the compositor, though this baseline could hide a small one.

Changes so far. Fixing a compile error at nsh-platform-macos/src/input.rs:1509, adding a way to load a caller-built EngineConfig (AppView::load_with_config, run_app_with_config), and a small desktop-shell entry. Rendering, session switching, sending, streaming and the command palette all worked.

Still open.

• Scoped and fallback key bindings: I only turned the capability flags on, so they don't fire yet (Ctrl+C).
• asset:/ images don't show (the sidebar logo). The image path probably doesn't use config.assets.
• Clicking sidebar rows in quick succession zoomed the window once. I haven't found why.
• The spike shell doesn't follow a replaced Host, restore window placement, reload UI, show notifications, pick folders or apply text scale.
• Not tested: real gesture feel, IME composition, or a window attached to 16, 256 and 1,024 running sessions.

Separately, both backends reach about 120 fps while streaming even though text arrives at 30 Hz, and GPUI redraws the whole window on each caret blink. I haven't traced the first one. It looks worth fixing whichever backend we use.

Next. I'd turn this into a draft PR against rust-preview: the embedding APIs, an opt-in AppKit path in desktop-shell, then the key bindings and assets. After that I'd run the many-session benchmark on both backends. @M4n5ter does that order work for you?

---

谢谢，这正是我接下来想验证的，所以已经跑了一轮。后面就基于 third-party/nsh 做。

结果： 真实工作台已经能跑在 AppKit 后端上，Host 和共享 TSX UI 都没动，我测到的场景开销都明显低于 GPUI。我建议继续在 macOS 上推进 AppKit，但现在还不能替换 GPUI：手势跟手感、输入法和一些功能缺口还没解决。

测试条件。 同一个 release 二进制，用环境变量选后端。Apple M5 Pro，内建 120 Hz 屏幕，窗口 1280×820（GPUI 为 1280×821）。窗口全程在前台，失去焦点的阶段直接作废。对话有 200 轮，包含 Markdown、代码块和表格。流式走真实 Host，模型是本地 OpenAI 兼容服务，每秒 30 个增量。

场景	GPUI CPU / GPU	AppKit CPU / GPU
静止	1.2% / 5%	0.0% / 1%
流式回复	15.1% / 32%	8.1% / 0%
滚动列表	26.6% / 37%	14.2% / 1%
三点，Animated.loop	29.2% / 43%	3.0% / 0%
三点，66ms 定时器	7.3% / 25%	1.4% / 0%
三点，125ms 定时器	5.0% / 19%	0.9% / 1%

CPU 是桌面壳进程占单核的比例，GPU 是全系统占用。每行是 2–6 段 5–15 秒的平均值，波动很小。三点动画在工作台上覆盖的面板里，面板有 32 行文字。滚动用工作台内 60 Hz 定时器调 scrollToOffset，速度 720 px/s，两边偏移一致，所以没有经过手势输入。注入滚轮事件我没能做到持续滚动。

之前的小探针，各跑一轮：原生动画 13.1% → 1.5%，15 Hz 定时器 2.5% → 0.3%，30 Hz 文字 6.1% → 2.3%，RSS 127 → 108 MB。

内存	GPUI	AppKit
RSS，静止 / 流式	232 / 239 MB	215 / 238 MB
footprint，静止 / 流式	280 / 286 MB	112 / 128 MB

RSS 差别不大，footprint 差别明显，GPUI 的部分包含 66 MB 的 IOSurface 和图形内存。这里没有计入 WindowServer 为 AppKit 图层持有的内存。

帧	GPUI	AppKit
流式：fps，帧间隔 p50 / p95	121，8.3 / 9.1ms	119，8.3 / 8.7ms
滚动：fps，帧间隔 p50 / p95	121，8.3 / 9.2ms	119，8.3 / 11.1ms

两个后端的计帧方式不同（GPUI 数绘制次数，AppKit 数有变化的刷新），只能粗略比较。节奏接近，但 AppKit 滚动时 p95 更宽，主线程最差一帧约 12ms，某一秒内最多有 4% 的帧超过一次刷新。我怀疑是挂载原生视图造成的，还没确认。这是我最想盯的风险。

WindowServer 在各场景都约 46–49%，主要是其他应用占着。滚动时 GPUI 只多 0.4 个点，AppKit 多 2.7 个点，看不出开销大量转移到合成器，但这个基线可能盖住小幅差异。

目前的改动。 修了 nsh-platform-macos/src/input.rs:1509 的编译错误，加了加载外部 EngineConfig 的入口（AppView::load_with_config、run_app_with_config），desktop-shell 加了一个小入口。渲染、切会话、发送、流式和命令面板都能用。

还没解决的。

• 带作用域和备用的快捷键：我只打开了能力开关，还不会触发（Ctrl+C）。
• asset:/ 图片不显示（侧栏 logo），可能是图片路径没用 config.assets。
• 快速连续点击侧栏时，窗口被放大过一次，原因没查到。
• 实验版的壳还没接 Host 切换、窗口位置、UI 重载、通知、选文件夹和文字缩放。
• 没测过：真实手势跟手感、输入法组字，以及窗口连着 16、256、1,024 个运行中会话的情况。

另外，两个后端流式时都跑到约 120 fps，而文字只有 30 Hz；GPUI 每次光标闪烁都重绘整个窗口。前者我还没查到原因，不管用哪个后端都值得修。

下一步。 我打算整理成一个 base 为 rust-preview 的 draft PR：先是嵌入接口和 desktop-shell 里可选开启的 AppKit 路径，再补快捷键和资源加载，然后在两个后端上跑多会话基准。@M4n5ter 这个顺序可以吗？

[M4n5ter]

Absolutely, it's definitely possible

[UncertaintyDeterminesYou4ndMe]

@Astro-Han thanks, these numbers settle the question for macOS. I'll wait for your draft PR so we don't work on the same files. Once it's up, feel free to assign us any part of the remaining list directly, for example bringing the AppKit shell to parity with the GPUI one (Host replacement, window placement, UI reload, notifications, folder picker, text scale), or checking IME, gestures and focus, which we went through on the GPUI backend in the last two weeks.

简体中文

@Astro-Han 谢谢，这组数据已经能说明 macOS 上该走 AppKit。我先等你的 draft PR，免得我们改到同一批文件。PR 出来后，剩下清单里有需要我们做的，直接指派给我们就行，比如让 AppKit 壳补齐 GPUI 壳已有的功能（Host 切换跟随、窗口位置、UI 重载、通知、选文件夹、文字缩放），或者验证输入法、手势和焦点，这几项我们前两周刚在 GPUI 后端上过了一遍。