Repository navigation
fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) - #497
mengmengmengqiang wants to merge 1 commit into
Conversation
|
Follow-up note (no action): after this PR the two waits are already separate constants — |
…Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460) A normal unmount whose connection outlives its mount waits for the receive loops to join. A healthy join follows the detach in milliseconds, but a connection kept alive by another mount namespace never joins, so the wait spends the whole drain_wait (5 s) before the Join stop is reported and the abort of Ephemeral-AI-Lab#496 can end it. Observe the join for its own shorter grace (JOIN_WAIT = 1 s) once every request is quiescent. Requests keep the whole drain_wait; only the loop join, which follows a known detach and can only finish in microseconds or never, is bounded shorter. A stalled unmount now ends in about 1.1 s instead of about 5.1 s; a healthy one is unchanged. Measurements on macOS 26.5.1 / OrbStack 7.2.9, the machine of Ephemeral-AI-Lab#496: - normal unmount: n=20, min 20 ms, p50 23 ms, p90 46 ms, max 82 ms - early-mount stall with drain_wait 5 s: n=20, 5036-5558 ms, p50 5075 - early-mount stall with JOIN_WAIT 1 s: n=20, 1021-1467 ms, p50 1082 - normal unmount with JOIN_WAIT 1 s: n=20, 17-44 ms, p50 23 - a 500 ms prototype (not this change) once saw a healthy join at 374 ms, which is why the grace is 1 s and not shorter. The stalled_unmount test also asserts the stalled unmount returns in under 3 s: it took 1.12 s with this change and 3.23 s with the full wait. Production LOC: 102831 -> 102833 (delta +2) core 38395 -> 38395 (0), engine 61777 -> 61779 (+2), api 2659 (0).
0bfd085 to
9a03429
Compare
|
感谢贡献,也感谢 AI 协作工具在定位 #426/#460 卡死路径时做的分析 🙏 真实评估(已读 diff): 对 AI agent 的使用价值:unmount 是 agent 自动化清理步骤的最后一环,这个改动把最坏卡住时间从 ~5 s 降到 ~1 s 且有确定边界。 建议:合入前在真实环境(有活跃 FUSE reader + mount namespace 副本)实测一次 unmount 全链路,把实测输出贴出来。功能性调整等 LayerFS 2.0(#446)发布后再做。 |
Rebased onto
mainafter #500 — #496 was reworked and merged as #500 (69fedcac), so this is now a single commit onmain(35d36575) and the file diff shows only this change.Problem
#496 ends a normal unmount stalled at
Joinwith the FUSE abort write, but it does so when the drain deadline (drain_wait, 5 s) expires. That deadline also covers requests, yet the join it then waits for follows a known detach: a healthy connection joins in milliseconds, while a connection kept alive by another mount namespace never joins (#426, #460). Paying 5 s for a wait that can only end in microseconds or never is the conservative part, not a necessary one.Change
Once every request is quiescent, observe the receive-loop join for its own shorter grace,
JOIN_WAIT = 1 s, instead of the remainingdrain_wait. Requests keep the wholedrain_wait; only the join is bounded shorter, and the abort of #496 then runs at ~1 s rather than ~5 s. Nothing else changes: the same abort write, the same second drain, the sameUnmounted.Measurements (same machine as #496: macOS 26.5.1, OrbStack 7.2.9, kernel
7.2.9-orbstack-00391, Docker 29.8.2)drain_wait5 s (#496)JOIN_WAIT1 s (this PR)JOIN_WAIT1 s (this PR)40/40 in the 1 s runs returned
UnmountedandClosed. The tail after the deadline (abort + force drain + revoke + close) was 36–558 ms, p50 75 ms, in the 5 s build.Why 1 s and not shorter: a throwaway 500 ms prototype (not in this PR) made a stalled unmount 519–565 ms, but one healthy unmount in that run took 374 ms — the normal tail moves under load, so 500 ms leaves ~1.3× margin. 1 s keeps ~2.7× over the slowest healthy join we saw and still cuts ~4 s per stall.
The signal to count is the grace path:
abort_attemptedin the follow-up event (draft: tokenaissance#2) is true exactly when the 1 s expired and #496's abort ran.Cost estimate for the scenario matrix
The 10 scripted scenarios contain 211
workspace.unmountcalls; #466's failing runs reported roughly 66 failed steps in one matrix pass. At ~4 s saved per stalled unmount, one full matrix run saves about 1–5 minutes, and a single agent call goes from ~5.1 s to ~1.1 s.Tests
stalled_unmountnow also asserts the stalled unmount returns in under 3 s (it took 1.12 s with this change, 3.23 s with the full wait); that assertion would fail on #496's build. The other daemon binaries are unchanged by this diff.Production LOC: 102831 -> 102833 (delta +2): core 0, engine +2, api 0.
Risk and rollback
The grace applies only after requests are quiescent, so it cannot cut off a running request; the worst case for a healthy-but-slow join is that it is treated as stalled and the abort ends a connection whose mount is already gone. Reverting is one line (
JOIN_WAITback todrain_wait), and the 5 s behaviour is #496.Visual evidence walk-through
The bug/fix board already shows both fixes: #496's recovery (now merged as #500) and this PR's grace. A one-picture guide first (what the bug is, what #496/#497 change, how to read the rest), then two context pictures (architecture, sequence), then the evidence: the fix board, the latency panel that isolates this PR's 1 s grace, and the options board that places it next to the alternatives. All measured on macOS 26.5.1, OrbStack (kernel
7.2.9-orbstack-00391), Docker 29.8.2.Start here. Left: the bug in one column. Middle: what #496 and #497 change, and what each costs. Right: how to read the rest of the pictures (lanes, numbered unmount steps, colors). All numbers in the middle column are measured on the machine named below.
Where this change lives. Core crates (content, history, storage) import no engine, FUSE, Docker or JavaScript code; the daemon/FUSE pair is in the adapter band.
JOIN_WAITis a timing constant inside that band — no architecture or dependency change.One bash call from create to unmount. The grace applies inside step 13: once requests are quiescent, the loop join is observed for 1 s instead of the remaining drain wait.
The left lane is the wedge that #496 recovers from; the right lane is this PR's
JOIN_WAIT = 1 s: requests keep the usual 5 sdrain_wait, then the receive-loop join is observed for only 1 s. Early-mount stall: n=20, 1021–1467 ms, p50 1082 (~1.08 s, down from ~5.1 s with #496 alone).The latency panel isolates the grace: a fast path (joined → return) and a stall path (grace expires → #496's abort). Normal unmount with the grace n=20 at 17–44 ms; without it n=20 at 20–82 ms.
The latency column is this PR, shown next to recovery, root-cause and rejected options. (The private-mount-namespace card is a separate experiment, now open as draft #499; its 22 ms is the prototype's early-mount measurement, not part of this PR.) The 1 s value leaves margin over the 374 ms healthy unmount measured under load, which is why 500 ms was rejected.