[upstream PR Ephemeral-AI-Lab#497] fix(fuse): bound the receive-loop join by a 1 s grace (Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460)

• Upstream: fix(fuse): bound the receive-loop join by a 1 s grace (#426, #460) Ephemeral-AI-Lab/layerfs#497
• Type: pull request
• State: OPEN
• Draft: False
• Mergeable: MERGEABLE
• Updated: 2026-10-28
• Upstream labels: -

Original body (excerpt)

Rebased onto main after Ephemeral-AI-Lab#500 — Ephemeral-AI-Lab#496 was reworked and merged as Ephemeral-AI-Lab#500 (69fedcac), so this is now a single commit on main (35d36575) and the file diff shows only this change.

Problem

Ephemeral-AI-Lab#496 ends a normal unmount stalled at Join with the FUSE abort write, but it does so when the drain deadline (drain_wait, 5 s) expires. That deadline also covers requests, yet the join it then waits for follows a known detach: a healthy connection joins in milliseconds, while a connection kept alive by another mount namespace never joins (Ephemeral-AI-Lab#426, Ephemeral-AI-Lab#460). Paying 5 s for a wait that can only end in microseconds or never is the conservative part, not a necessary one.

Change

Once every request is quiescent, observe the receive-loop join for its own shorter grace, JOIN_WAIT = 1 s, instead of the remaining drain_wait. Requests keep the whole drain_wait; only the join is bounded shorter, and the abort of Ephemeral-AI-Lab#496 then runs at ~1 s rather than ~5 s. Nothing else changes: the same abort write, the same second drain, the same Unmounted.

Measurements (same machine as Ephemeral-AI-Lab#496: macOS 26.5.1, OrbStack 7.2.9, kernel 7.2.9-orbstack-00391, Docker 29.8.2)

40/40 in the 1 s runs returned Unmounted and Closed. The tail after the deadline (abort + force drain + revoke + close) was 36–558 ms, p50 75 ms, in the 5 s build.

Why 1 s and not shorter: a throwaway 500 ms prototype (not in this PR) made a stalled unmount 519–565 ms, but one healthy unmount in that run took 374 ms — the normal tail moves under load, so 500 ms leaves ~1.3× margin. 1 s keeps ~2.7× over the slowest healthy join we saw and still cuts ~4 s per stall.

The signal to count is the grace path: abort_attempted in the follow-up event (draft: #2) is true exactly when the 1 s expired and Ephemeral-AI-Lab#496's abort ran.

Cost estimate for the scenario matrix

The 10 scripted scenarios contain 211 workspace.unmount calls; Ephemeral-AI-Lab#466's failing runs reported roughly 66 failed steps in one matrix pass. At ~4 s saved per stalled unmount, one full matrix run saves about 1–5 minutes, and a single agent call goes from ~5.1 s to ~1.1 s.

Tests

stalled_unmount now also asserts the stalled unmount returns in under 3 s (it took 1.12 s with this change, 3.23 s with the full wait); that assertion would fail on Ephemeral-AI-Lab#496's build. The other daemon binaries are unchanged by this diff.

Production LOC: 102831 -> 102833 (delta +2): core 0, engine +2, api 0.

Risk and rollback

The grace applies only after requests are quiescent, so it cannot cut off a running request; the worst case for a healthy-but-slow join is that it is treated as stalled and the abort ends a connection whose mount is already gone. Reverting is one line (JOIN_WAIT back to drain_wait), and the 5 s behaviour is Ephemeral-AI-Lab#496.

Visual evidence walk-through

The bug/fix board already shows both fixes: Ephemeral-AI-Lab#496's recovery (now merged as Ephemeral-AI-Lab#500) and this PR's grace. A one-picture guide first (what the bug is, what Ephemeral-AI-Lab#496/Ephemeral-AI-Lab#497 change, how to read the rest), then two context pictures (architecture, sequence), then the evidence: the fix board, the latency panel that isolates this PR's 1 s grace, and the options board that places it next to the alternatives. All measured on macOS 26.5.1, OrbStack (kernel 7.2.9-orbstack-00391), Docker 29.8.2.

_{Start here. Left: the bug in one column. Middle: what Ephemeral-AI-Lab#496 and Ephemeral-AI-Lab#497 change, and what each costs. Right: how to read the rest of the pictures (lanes, numbered unmount steps, colors). All numbers in the middle column are measured on the machine named below.}

_{Where this change lives. Core crates (content, history, storage) import no engine, FUSE, Docker or JavaScript code; the daemon/FUSE pair is in the adapter band. JOIN_WAIT is a timing constant inside that band — no architecture or dependency change.}

_{One bash call from create to unmount. The grace applies inside step 13: once requests are quiescent, the loop join is observed for 1 s instead of the remaining drain wait.}

_{The left lane is the wedge that Ephemeral-AI-Lab#496 recovers from; the right lane is this PR's JOIN_WAIT = 1 s: requests keep the usual 5 s drain_wait, then the receive-loop join is observed for only 1 s. Early-mount stall: n=20, 1021–1467 ms, p50 1082 (~1.08 s, down from ~5.1 s with Ephemeral-AI-Lab#496 alone).}

_{The latency panel isolates the grace: a fast path (joined → return) and a stall path (grace expires → Ephemeral-AI-Lab#496's abort). Normal unmount with the grace n=20 at 17–44 ms; without it n=20 at 20–82 ms.}

_{The latency column is this PR, shown next to recovery, root-cause and rejected options. (The private-mount-namespace card is a separate experiment, now open as draft Ephemeral-AI-Lab#499; its 22 ms is the prototype's early-mount measurement, not part of this PR.) The 1 s value leaves margin over the 374 ms healthy unmount measured under load, which is why 500 ms was rejected.}

Sync note

This is a read-only mirror. The upstream PR is the source of truth; no code/branches are copied.