docs(512): propose a VM Sandbox backend (Sandboxes as namespaces in one VM per Store) by yifanxuaaa · Pull Request #515 · Ephemeral-AI-Lab/layerfs

yifanxuaaa · 2026-10-30T03:43:09Z

Proposal only, for #512. Nothing is implemented, and owner decisions V1–V10 are open.

What it proposes (core/docs/issues/512/README.md):

One Linux VM per Store. Inside it, the existing worker runs the SDK with the Store opened in-process on the VM's ext4 disk, which is today's Linux topology.
A namespace backend replaces Docker inside the VM:
- each Sandbox gets its own mount, PID and IPC namespaces and a cgroup;
- its daemon runs as root with /dev/fuse;
- commands run as a nonroot uid:gid with no capabilities and no-new-privs.
It uses safe nix calls from one small helper binary, with no unsafe Rust and no util-linux.
Provider adapters in @layerfs/runtime. Each one creates the VM, attaches the Store disk, starts the worker and opens one private stream to it. The first candidates are Microsandbox and any Linux host over SSH, such as EC2.
Docker stays the default. The new backend is a second variant of a concrete Runtime enum.

Evidence: the #512 experiment, 2026-10-30.

Every local environment hosted real LayerFS.
A test-only stand-in ran the unchanged SDK and daemon with no Docker in an Apple Container VM; all 15 calls passed.
Every platform needs a user-namespace block; none blocks user namespaces by default.

Decisions needed (§10):

the shape (V1);
widening the worker route to VM placements from any host OS (V2);
which adapters first (V3);
the sysctl or a seccomp filter for user namespaces (V4);
per-Sandbox networking (V5);
nix features (V6);
the image format (V7);
Store durability (V8);
setting names (V9);
whether to keep the stand-in (V10).

Production LOC: 102831 -> 102831 (delta 0), docs only.

🤖 Generated with Claude Code

Add core/docs/issues/512/README.md: a proposal, not implemented, for a second Sandbox backend. Each Store gets one Linux VM; inside it the existing worker runs the SDK with the Store in-process, and a namespace backend replaces Docker (mount/PID/IPC namespaces, a cgroup per Sandbox, commands as a nonroot uid with no capabilities and no-new-privs). Provider adapters in @layerfs/runtime create the VM and open one private stream to the worker. Docker stays the default. Evidence is the #512 experiment; owner decisions V1-V10 are open. Production LOC: 102831 -> 102831 (delta 0) Scope and method: tools/production_loc.py at parent 35d3657 and at the staged tree (docs only). Subtotals unchanged: core 38395, engine 61777, api 2659; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…#515) Add the `vm:<provider>` placement to @layerfs/runtime (LAYERFS_PLACEMENT, or the `placement` setting; `local` stays the default and is unchanged): one Linux VM per home holds every project's Store on a named ext4 disk volume, serves the home's worker, and hosts the Sandboxes as Linux namespaces (LAYERFS_SANDBOX_BACKEND=namespaces). The host reaches only the worker's loopback-published port, with one attempt per call and no replay. The adapter (api/runtime/src/platform/vm/) attempts each of Ephemeral-AI-Lab#515 §6's five steps once: ensure the VM exists and runs (create or start, with the Store disk and a bind mount of the home); copy the sealed bundle in once per bundle hash; block user namespaces (V4, written and read back) and start the worker as root with its secret in a 0600 guest file, never in a command line; write the endpoint useWorker reads from the VM's private port (V5: Sandboxes share the VM's network, so daemon ports stay inside); and stop or delete the VM (V8: a delete is refused while any project of the home exists, because its Store is on that disk and nowhere else and this build has no remote Store push). Providers: microsandbox (`msb`, verified end to end in a real microVM) and apple-container (`container`, UNVERIFIED: that CLI is not installed on the machine this was written on; its inspect mapping refuses a shape it does not know instead of guessing, so a changed CLI is an error and never a VM the adapter believes is running). Runtime changes: four Store hooks (makeStore, storeHolders, releaseStoreHolders, removeStore) replace the inline Docker calls of project init and removal, with byte-identical Docker implementations for the local platforms; status() reports `placement`, the VM and `docker: { level: "not-used" }`; the Docker-shaped helpers refuse a VM home by name (volume.list/remove, sandbox.list, list_leftovers, clean_leftovers) and sweep answers `{ host: false }`; runtime.vm adds status/stop/delete for a VM home only. Also: the SDK client reads LAYERFS_SANDBOX_BACKEND once and sends it with every call, so a Sandbox made with the namespaces backend is not reopened as the other local backend (a real VM refused exactly that on project.commit); and the Linux sealed bundle gains layerfs-sandbox-helper, which the namespaces backend needs beside the daemon (and LAYERFS_SANDBOX_HELPER for a Sandbox reopened from its record). Tests: api/runtime/test/vm_placement.mjs (settings, the five steps, bundle hash, V8 delete, host-Docker refusals, apple-container's flags; stand-in provider CLIs), api/runtime/test/verify/vm_real.mjs (a real microVM: Init through the VM worker, a namespace Sandbox, exec as 501, the V4 block, a Commit, the V8 refusal, VM stop/start, a worker restart, a second isolated Sandbox), the existing api/runtime tests 14/14, and the macOS smoke 48/48 with placement unset. Production LOC: 111921 -> 112431 (delta +510). Scope: first-party production source under core/, engine/ and api/ (each api package's src/), counted by tools/production_loc.py; tests, docs, tools and node_modules excluded. api 2681 -> 3191; core and engine unchanged. Reference: absent (retired R9c; last counted 65417 at tree 498dd19).

Strictly follow proposal Ephemeral-AI-Lab#515's "one Linux VM per Store" (V1): - Each project gets its own Linux VM (<prefix><project>-vm), its own Store disk (<prefix><project>-vm-store), host port, bundle copy, and worker secret. - runtime.project.start_init creates the project's VM and disk; runtime.project.remove removes both together. - runtime.worker.start(project) and restart(project) act on that project's VM. - runtime.vm.status(project?), stop(project), delete(project) operate per project; status() lists every project VM with its state, worker and disk free bytes. - useWorker widen to accept an endpoint resolver (project => file) so one client process can direct Store calls to each project's VM worker. - deploymentEnv sets WORKER_ENDPOINT for subprocess agents to that project's VM. - Reserve V7 bundle Sandbox image paths (LAYERFS_IMAGE / LAYERFS_IMAGE_ARCHIVE) for forthcoming H38 namespaces backend merge. - Verified against Apple Container 1.6.1 (aa8d76b) on macOS 27.2: fixed inspect JSON array output (no --format flag), volume sizing, -i on exec stdin, and remount /proc/sys rw for V4 user namespace sysctl block. E2E verified on both microsandbox and apple-container real VMs. Production LOC: 112433 -> 112509 (delta +76) - core: 41400 -> 41400 (+0) - engine: 67840 -> 67840 (+0) - api: 3193 -> 3269 (+76) - reference: absent (retired R9c; last counted 65,417 at tree 498dd19) - combined: 112433 -> 112509 (delta +76)

…meral-AI-Lab#515) The `vm:<provider>` placement now runs the namespaces backend the way Ephemeral-AI-Lab#515 §4.7 V7 (H38) requires: the Sandbox root is the bundle's docker-archive image, its per-Sandbox overlay upper must sit on a local filesystem, and the client a VM's agent process runs as is given the image's digest *and* its archive. - `settings.js`: `GUEST.local` is `/layerfs/store/local`, on the project VM's ext4 Store disk instead of its root. Microsandbox's guest root is an overlay (`stat -f` / mountinfo: `overlay` over `/.msb/rootfs`), which `require_overlay_upper` refuses as an overlay upper, so every namespaces Sandbox in a Microsandbox VM would have refused to start; Apple Container's root is ext4, and the Store disk is ext4 on both. The Store's own directory (`/layerfs/store/global`) stays outside it, as `check_store` requires. - `platform/vm/index.js`: `deploymentEnv` also gives clients `LAYERFS_IMAGE_ARCHIVE` (the guest path of this VM's own bundle copy), which the namespaces backend needs beside `LAYERFS_IMAGE`; the worker already had both. Docs record the guest layout and that the image is required. - Tests: `vm_placement.mjs` (flow) asserts the archive path and the Store-disk Sandbox directory. `vm_real.mjs` proves, on both providers: the command's root is the image (`/etc/os-release`, the marker, no `/bin/node`, the VM's `/layerfs` not visible), the image's directories are not writable by the command while its own layer is, two Sandboxes' writes stay each one's, one digest is unpacked once (cache inode/mtime unchanged, mode 555, no `.tmp-*`), `LAYERFS_LOCAL_DIR` is on ext4, and a VM restart leaves a stopped Sandbox record that is not reopened onto a new root, is reclaimed by the next `sandbox.create` (its number returns) and runs from the same unpack. Merge: this branch also merges `h38-oci-rootfs` (27d92c5) with `--no-ff` (commit 9c8c1e0); no file conflicted, so there is no conflict resolution to record. One integration gap surfaced there and is fixed here (the archive path above) plus the `/layerfs/local` -> Store-disk move that H38's `require_overlay_upper` makes necessary in a real VM. Production LOC (tools/production_loc.py, nonblank non-comment first-party product source; tests, docs and tools excluded): vs H37b head 06ae574 (the branch's first parent's tree before the merge): core 41,400 -> 41,400 (delta 0) engine 67,840 -> 69,037 (delta +1,197) H38's namespaces rootfs work api 3,269 -> 3,276 (delta +7) H38's image-archive deployment reference absent (retired R9c; last counted 65,417 at tree 498dd19) combined 112,509 -> 113,713 (delta +1,204) vs H38 base bdc341a: core 41,400 -> 41,400 (delta 0) engine 67,840 -> 69,037 (delta +1,197) api 2,681 -> 3,276 (delta +595) H37b's per-Store VM work reference absent (retired R9c; last counted 65,417 at tree 498dd19) combined 111,921 -> 113,713 (delta +1,792) this commit alone (9c8c1e0 -> here): api 3,276 -> 3,276 (delta 0): a path value and one more field on an existing line; comments and tests are excluded. Checks: cargo fmt --check; clippy --all-targets -D warnings (macOS); clippy --target x86_64-unknown-linux-gnu -p layerfs-sandbox -D warnings; tools/check_product_boundary.py PASS (1029 files); tools unittest 155 OK; api/runtime/test/runtime.mjs 14/14; vm_placement.mjs 7/7; api/agent/test/packaging.mjs 8/8 (one case's `await using` spelled out for Node 22's parser in a temporary copy; the file is unchanged); macOS smoke 48/48 with this tree's release binaries; vm_real.mjs microsandbox 12.3 s and apple-container 47.5 s, both ok.

…meral-AI-Lab#515) Adds api/runtime/src/platform/vm/ssh.js so a LayerFS home can be placed on an existing Linux host reachable over SSH (EC2, local VM, or bare metal; Ephemeral-AI-Lab#515 §6, V3): - platform/vm/ssh.js: implements the five adapter steps over SSH with the system ssh client. The host kernel is verified (uname -s is Linux), the user-provided Store disk is checked (findmnt or blkid is ext4) and attached to /layerfs/store, the bundle is uploaded via tar streaming, the worker runs as root with its secret and user namespaces blocked, and the private stream is an ssh -L port forward with -o ExitOnForwardFailure=yes. StrictHostKeyChecking defaults to yes; key-only authentication via identity file paths (no passwords, no raw keys in argv/logs). One host serves one Store (refuses Busy on collision). Stops and deletes clean up the forward, bundle, and worker state while preserving the user's Store disk. - platform/vm/apple-container.js: cleans up newly created named Store volumes if create fails (e.g. port conflict), fixing the orphan volume issue from H39v. - platform/vm/microsandbox.js: cleans up ~/.microsandbox/sandboxes/<name> if create fails, fixing the hidden state issue from H39v. - settings.js: registers vm:ssh provider, validates host, user, port, identityFile, storePath, and strictHostKeyChecking settings. Missing host is reported in missing(). - Tests: vm_placement.mjs adds ssh and cleanup suites (9/9 pass). New test/verify/ssh_faults.mjs verifies the complete failure matrix (port collision, missing store path, non-ext4, non-Linux, unreachable drop, disconnect & reconnect, remote host reboot & reuse, delete with and without unpushed Store, shared host collision). All 9 scenarios pass. Real E2E runs against live Linux VM over SSH (7.46 s total, agent phase 1.26 s), verifying workspace scenario parts, image root read-only, two Sandboxes isolated, single unpack reuse, and ext4 filesystem. Production LOC (tools/production_loc.py, nonblank non-comment first-party product source; tests, docs and tools excluded): vs 5d9dc29 (HEAD of h40-ssh): core 41,400 -> 41,400 (delta 0) engine 69,037 -> 69,037 (delta 0) reference absent (retired R9c; last counted 65,417 at tree 498dd19) api 3,276 -> 3,619 (delta +343) dsh 0 -> 0 (delta 0) combined 113,713 -> 114,056 (delta +343) Crate breakdown (api scope): api/runtime/src/platform/vm/ssh.js +284 (new) api/runtime/src/settings.js +32 (85 -> 117) api/runtime/src/platform/vm/apple-container.js +13 (78 -> 91) api/runtime/src/platform/vm/microsandbox.js +12 (60 -> 72) api/runtime/src/platform/vm/index.js +2 (310 -> 312) Checks: cargo fmt --check; cargo clippy --workspace --all-targets --no-deps (0 warnings); tools/check_product_boundary.py PASS (1030 files); tools unittest 56 OK; api/cli/test/cli.test.mjs 75/75 PASS; api/runtime/test/runtime.mjs 14/14 PASS; api/runtime/test/vm_placement.mjs 9/9 PASS; api/runtime/test/verify/ssh_faults.mjs 9/9 PASS; vm_real.mjs microsandbox 12.1 s and apple-container 51.0 s, both OK; real Linux VM E2E over SSH (e2e-ssh.mjs) 7.46 s, all checks PASS.

…der (Area B1) Extract the complete Sandbox runtime interface and shared native/node plumbing from 9aa10e6 onto upstream/main (35d3657), implementing the Docker backend and leaving same-signature Unsupported slots for backend/{linux,macos,namespaces}/ so B2, B3 and #515 only replace their own backend directories. In layerfs-sandbox: - Define concrete Runtime (Docker, MacosNative, LinuxNative, Namespaces, Windows) and Sandbox (Docker, Native, Namespaces) enums in backend/runtime.rs. - Add provider capabilities (including LoopbackNfs, MicroVm, RemoteCloud), LaunchPhase/LaunchFailure, ProviderLaunchOptions (Local, Namespaces, Docker) and SandboxRecord (Local, Namespaces, Docker). - Add types::Image and cross-backend ExecRequest; move shared json and strings helpers to backend/. - Implement DockerProvider and DockerInstance over existing Docker Engine lifecycle and streaming execution. - Include shared backend/native/* lifecycle, identity, listener and streams modules under cfg(any(linux, macos)), with same-signature Unsupported stubs in backend/{linux,macos,namespaces}/. In layerfs-api: - Route SandboxApi through Arc<Runtime> via with_runtime and create_provider_file, retaining local resource ownership and command identity validation. - Share TCP/Noise/Hello and Store install pipeline in sandbox/connection.rs. In layerfs-node and @layerfs/agent: - Take verbatim 9aa10e6 shared binding modules (local, discovery, runtime, sandbox, workspace, exec, registry, lib) with a no-op unmount slot for B3. - Expose deploy.sandbox, deploy.localDir, deploy.sandboxBackend and deploy.imageArchive settings in @layerfs/agent. Production LOC: 102831 -> 106022 (delta +3191) First-parent comparison: upstream/main 35d3657 (tree ddea2b5) -> committed tree ae48e29. Scopes: - core: 38395 -> 38395 (delta 0) - engine: 61777 -> 64938 (delta +3161) * layerfs-sandbox: 3417 -> 5526 (delta +2109) * layerfs-api: 1940 -> 2081 (delta +141) * layerfs-node: 2744 -> 3655 (delta +911) - api: 2659 -> 2689 (delta +30) * @layerfs/agent: 849 -> 879 (delta +30) - dsh: 0 -> 0 (delta 0) - reference: absent (retired R9c; last counted 65,417 at tree 498dd19) Method: tools/production_loc.py (blob 3341e4d), --root <exact exported tree> --json for both snapshots; first-party production Rust, runtime SQL and JavaScript; tests, docs, tooling, generated and third-party files excluded.

VM Sandbox backend (#512 proposal, #515): one Linux VM per project whose Sandboxes are Linux namespaces (namespace backend, image root, the layerfs-sandbox-helper binary), with Microsandbox, Apple Container and SSH providers in @layerfs/runtime. Merged at PR head 8724090 onto the B3 merge a1cd85c. The PR predates #506 and #511 on this branch; the merge ports it: - project/remove.js: #511 replaced it with project/delete.js and #506 moved its volume steps into store/volume.js removeVolume. The VM's four platform Store hooks (storeHolders, releaseStoreHolders, releaseStore, removeStore) now run inside removeVolume, which takes `{ name, volume }`; delete.js, store calls.js and sync.js pass it. On local placements the shared hooks issue the same Docker commands as before; Windows' releaseStore reads `{ volume }` as the PR made it. remove.js stays deleted. - clients.js: a platform's own deploymentEnv (a `vm:` home) comes first, then v2's PostgreSQL environment, then the Docker default. - sandbox leftovers/list/sweep and volume.js: the VM's no-host-Docker guards come first, then v2's shared-Store aware code; both imports kept. - Tests: verify/vm_real.mjs calls project.delete (kind "Deleted"); the namespaces Linux tests give their Docker records `Some(...)` volumes. Gaps: a `vm:` home's deploymentEnv does not cover PostgreSQL Stores, and v2's shared-Store calls (store.create, load) still make host Docker volumes, so neither is wired to VM placements. Verified on macOS (Rust 1.85.1): workspace warning-denying Clippy, all targets; format check (only the existing minio.rs drift); layerfs-sandbox tests 56 passed, 0 failed, 1 ignored (namespaces tests are Linux-only). @layerfs/runtime with the worktree's own agent/backend linked in a temporary node_modules (removed after): vm_placement all 9 selections PASS (settings, flow, two, bundle, delete, hosts, apple-container, ssh, cleanup); runtime.mjs all 15 selections PASS. Boundary guard PASS with self-tests; tools/test_worktree.py OK; Ruff 0.17.2 format check clean. Not run: Linux build and tests (no Linux C toolchain on the host, Docker not running), verify/vm_real.mjs and ssh_faults.mjs against real VMs. Disposable only; Durable NOT_RUN, disabled by owner. Production LOC: 119822 -> 123141 (delta +3319) First-parent comparison: layerfs_v2_0_0 a1cd85c (tree 63d9a3c) to prepared merge tree 01a2878. Scopes: core 41405 -> 41405 (+0), engine 75122 -> 77512 (+2390), api 3295 -> 4224 (+929). By crate: layerfs-sandbox +2390 (namespace backend and the layerfs-sandbox-helper binary), @layerfs/runtime +924, @layerfs/agent +5. Method: tools/production_loc.py (blob 686d269) --root <exported tree> --json for both snapshots; first-party production Rust, runtime SQL and JavaScript; tests, docs, tooling, generated and third-party files excluded. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Deleted tests: none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

yifanxuaaa · 2026-10-31T18:27:58Z

Closing: the VM Sandbox backend this proposes is implemented in #526, which is now merged into layerfs_v2_0_0 (merge commit 81f8758). This proposal document itself (core/docs/issues/512/README.md) was not merged.

This was referenced Oct 30, 2026

Sandbox backends: which agent sandboxes can replace LayerFS's Docker container? Local stand-ins first #512

Open

Sandbox backends: twelve-cell speed comparison, Docker vs the no-Docker VM route (sub-issue of #512) #516

Open

jiangkoumo mentioned this pull request Oct 30, 2026

feat(sandbox): 用具体的 Runtime enum 统一 Docker / Linux 原生 / macOS 原生，并新增 Linux namespaces 后端（B1/B2/B3，按 #515，叠在 #427 上） #498

Closed

yifanxuaaa closed this Oct 31, 2026

yifanxuaaa mentioned this pull request Oct 31, 2026

Sandbox platform integration and consolidation #527

Open

Navigation Menu

Repository navigation

docs(512): propose a VM Sandbox backend (Sandboxes as namespaces in one VM per Store) - #515

docs(512): propose a VM Sandbox backend (Sandboxes as namespaces in one VM per Store)#515
yifanxuaaa wants to merge 1 commit into
mainfrom
issue512-vm-backend

yifanxuaaa commented Oct 30, 2026

Uh oh!

yifanxuaaa commented Oct 31, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

yifanxuaaa commented Oct 30, 2026

Uh oh!

yifanxuaaa commented Oct 31, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant