Repository navigation
Sandbox backends: which agent sandboxes can replace LayerFS's Docker container? Local stand-ins first #512
Description
Status (2026-10-30)
Every environment hosted real LayerFS. Route A (no Docker at all) passed end to end in Apple Container, Microsandbox and Firecracker. The proposed backend is in draft PR #515.
| Environment | FUSE + privilege split | Store | Real LayerFS | Comments |
|---|---|---|---|---|
| Microsandbox | pass | per-VM ext4 disk; survives VM deletion | route B pass; route A pass | results, route A |
| Daytona OSS (local) | pass | loop ext4 inside the sandbox | route B pass (arm64 runner patched locally) | iteration 1, iteration 2 |
| Apple Container | pass with --cap-add SYS_ADMIN |
named ext4 volume, one VM per project | route B pass; route A pass | iteration 1, iteration 2, route A |
| Lima VM (EC2 stand-in) | pass | ext4 | route B pass | results |
| Firecracker guest (E2B/Vercel stand-in) | pass | second ext4 drive | route A pass; route B probe only | probes, route A |
What holds everywhere:
- One VM (or one privileged sandbox) per project, not per Sandbox. VMs don't share a kernel, so the Store goes on the project VM's own ext4 disk, and its daemons run inside it, each in its own mount namespace.
- Commands through
setpriv(uid:gid, no capabilities, no-new-privileges). Unmount is denied. - User namespaces must be blocked by the backend. No platform blocks them by default. Use
user.max_user_namespaces=0in a per-project VM, or a seccomp filter in the command launcher where the kernel is shared, as on Daytona. - The daemon endpoint can be reached privately on each platform: a loopback port forward, the SSH gateway, an exec stdio bridge, or vsock.
Open: hosted providers' own kernels and policies; a real backend in place of the test-only stand-in (provider adapter, persistence of backend state, seccomp and cgroups); the SDK outside the VM (the worker route).
Goal
LayerFS should be able to run on more than one Sandbox backend. Today it has one, Docker (core/crates/layerfs-sandbox/src/backend/docker/). This experiment finds out which agent-sandbox platforms could become a backend that replaces the Docker container and does every SDK operation the Docker backend does today. Start with environments that run on a developer Mac, then move to the hosted providers they stand in for.
(Revised 2026-10-30, same day: the first version asked mainly whether LayerFS's current Docker containers could run nested inside a provider. That is now a secondary route.)
Background
Today, for each Sandbox LayerFS starts one Docker container. Its daemon mounts the Workspace inside it and the agent's commands run there too. The container gives LayerFS:
- a real Linux kernel with FUSE: the daemon opens
/dev/fuseand callsmount(2)itself (core/crates/layerfs-fuse/src/session/startup.rs); - a privilege split: the daemon runs as root with
CAP_SYS_ADMINand mounts withallow_other, while commands run as a separate nonroot uid:gid with no capabilities,no-new-privileges, and a seccomp filter that denies user namespaces. So a command cannot unmount the Workspace or copy the mount into a namespace of its own (container.rs,topology.rs,core/docs/architecture/76-native-mount-session.md); - the Store on a local ext4, xfs or btrfs volume, shared by the project's daemons on one kernel. On Linux, SDK Store calls also run in the caller's own process.
Two routes per platform
- Route A, a new backend (the main question): the platform's sandbox is the Sandbox. The daemon mounts directly in it and there is no Docker. This needs new backend code, and the platform must provide the privilege split and every operation in the contract below.
- Route B, nested Docker (secondary): LayerFS runs unchanged and starts its usual container inside the platform's VM. No code changes, but the platform must allow Docker inside with these flags.
Desk research (2026-10-30)
| Provider | Finding |
|---|---|
| EC2 | the only confirmed fit |
| E2B, Vercel Sandbox, Modal VM runtime, Daytona Linux VM | likely fits: microVMs with root and documented FUSE |
| Fargate, Lambda, Azure Container Apps, Modal's default gVisor runtime | not viable |
The Mirage project (strukto-ai/mirage) needs the same --cap-add SYS_ADMIN --device /dev/fuse for container mounts.
The backend contract (what the Docker backend does at HEAD)
| SDK call | Backend steps |
|---|---|
sandbox.create |
B1 create, B2 attach Store volume, B3 upload, B4 start, B5 startup log, B6 endpoint |
workspace.mount / commit / status / unmount |
daemon calls over B6's TCP endpoint (no backend step) |
workspace.exec |
B7 exec |
sandbox.get_log |
B5 log tail |
SDK restart, sandbox.reclaim |
B8 reopen by ID, B10 discover |
sandbox.close |
B9 stop then delete, Store volume kept |
| Store calls (fs, branch, project) | S1: open the Store file directly from a Linux process on the same kernel |
Each platform is checked item by item:
- B1 Create from a pinned image (Docker uses
sha256:digest) or an
equivalent pinned template, with labels; one acknowledged full unique ID; no
adoption by name. - B2 Shared Store volume: one persistent local ext4/xfs/btrfs volume,
read-write at a fixed path (/layerfs-store), attached to several sandboxes
of one project at the same time on ONE Linux kernel (SQLite locking plus a
Linux directory lock). Record the filesystem type and whether two sandboxes
can share it. If each sandbox is its own VM, they cannot share a kernel:
record that, and whether several daemons can run inside one VM instead. - B3 Upload the daemon executable (0755) and private config (0600) before
start (Docker: archive into the stopped container). A file upload API, or a
pinned image that already contains them. - B4 Start the daemon as root with CAP_SYS_ADMIN and /dev/fuse, an init
that reaps orphans, and the privilege split inlfs-route-a-probe.sh. - B5 Startup log: read the daemon's stdout stream to see its listener
marker; a bounded tail of its log after a failed start. - B6 Endpoint: the daemon's TCP port reachable from the SDK host. Docker
publishes it on host 127.0.0.1 only. Record how (port forward, tunnel,
preview URL) and whether it can stay private (not public internet). - B7 Exec: non-TTY, as a given nonroot uid:gid, argv with no shell, env,
working directory = the mount, optional stdin with EOF, stdout and stderr as
separate streams, exit code, a unique exec ID. Docker has no per-Exec
cancel (it returns Unsupported): record whether the provider can kill one
exec (a gain over Docker). - B8 Reopen an existing sandbox by exact ID after the SDK process
restarts; see running vs stopped; read its configuration to check it. - B9 Stop and delete one sandbox; the Store volume survives.
- B10 Discover sandboxes by label or name prefix.
- S1 Store access: the SDK opens the Store file in its own process on
Linux. With a remote provider the SDK runs elsewhere, so this needs either
the SDK running inside the provider VM or the existing worker route (a
Linux worker beside the Store, reached over a private authenticated port).
Record which is possible.
Environments
| Environment | Stands in for | Notes |
|---|---|---|
| Daytona open source v0.190.0, local Docker Compose | Daytona cloud container sandboxes | The repo was archived in June 2026. Its sandboxes are privileged containers inside a privileged runner with its own Docker, so a pass here is a best case: the cloud likely runs a different container runtime (the repo mentions Sysbox) |
| Microsandbox (libkrun, Apple Silicon) | E2B, Vercel and Modal VM-style microVMs | Also records how virtio-fs volumes look |
| Apple Container (macOS 26) | VM-per-container platforms | --cap-add SYS_ADMIN |
| Lima VM with nested virtualization, then Firecracker | a plain Linux VM such as EC2, then E2B and Vercel (both Firecracker) | Needs /dev/kvm inside Lima |
Levels
- L1, FUSE in the environment (
lfs-sandbox-probe.sh):/dev/fuse, capabilities,no_new_privs, seccomp, user namespaces, privilege to mount, a real FUSE mount (fuse-overlayfs), the filesystem of/and/tmp. - L2, the privilege split (
lfs-route-a-probe.sh): root mounts withallow_other; a capability-less uid-1500 command reads and writes the mount but cannot unmount it; whether a command can create a user namespace and copy the mount, and whether root can block that (for exampleuser.max_user_namespaces=0). - L3, the backend contract: B1–B10 and S1 against the platform's own API, each yes / no / how.
- L4, real LayerFS where L1–L3 allow: the sealed arm64 Linux build
build-linux-bb5f636bcbca4078a0d7a5368b3e101b(source7ab54bcf1a). Init a project, mount a Workspace, run a command through it, Commit, and unmount. - Route B, optional:
lfs-sandbox-probe.sh's nested-Docker step with LayerFS's exact flags.
Reference run of the L2 probe on Docker Desktop with seccomp=unconfined, which is how a plain VM looks: the mount, the command's read and write, and the unmount denial all pass, but cmd_userns=allowed and cmd_copy_mount=SUCCEEDED. With no Docker filter, the backend must block user namespaces some other way.
Rules
- One attempt per step, no retries.
- Every command bounded at 2 minutes, and never more than 5 for a single large pull.
- Disposable Stores only, created for the experiment. Never the owner's Stores, containers or volumes.
- Each environment's own containers, VMs and volumes are removed afterwards.
- Installed tools and pulled images are listed in the results.
Results
Per environment, this issue gets one comment with the probe outputs, an L1–L4 verdict, the contract table, and what the result does and does not prove.
- Daytona OSS (local)
- Microsandbox
- Apple Container
- Lima, then Firecracker
- Route A end to end (test-only Engine API stand-in, no Docker)
L2 probe (lfs-route-a-probe-v2.sh, version 2: error files go in a directory the invoking shell owns)
#!/bin/sh
# LayerFS route-A probe: can this environment REPLACE the Docker container,
# i.e. host the daemon's FUSE mount and LayerFS's privilege split with no
# Docker? Prints KEY=VALUE lines. Writes only under /tmp/lfsrouteA, and adds a
# throwaway user `lfsprobe` (uid/gid 1500) if missing.
#
# LayerFS today (Docker backend): the daemon runs as root with CAP_SYS_ADMIN
# and /dev/fuse and mounts with allow_other; commands run as a separate nonroot
# uid:gid with no capabilities, no-new-privileges, and a seccomp filter that
# forbids user namespaces, so a command cannot unmount the Workspace or copy
# the mount into a namespace of its own. Route A needs the same split here.
#
# Needs root or passwordless sudo. Each step is one attempt with a timeout.
# Usage: sh lfs-route-a-probe.sh (PROBE_INSTALL=1 installs fuse3 +
# fuse-overlayfs + util-linux via apt/apk if missing)
set -u
W=/tmp/lfsrouteA
say() { printf '%s=%s\n' "$1" "$2"; }
T() { if command -v timeout >/dev/null 2>&1; then timeout "$@"; else shift; "$@"; fi; }
SUDO=""
if [ "$(id -u)" != 0 ]; then
if sudo -n true 2>/dev/null; then SUDO="sudo -n"; else say route_a "skipped: needs root or passwordless sudo"; exit 0; fi
fi
say route_a_version 2
# error and output files are written by this (maybe nonroot) shell, so they go
# in a directory it owns, not under the root-owned $W
E=$(mktemp -d)
$SUDO mkdir -p "$W/src/pub" "$W/up" "$W/wk" "$W/mnt" "$W/copy"
echo "probe-file" | $SUDO tee "$W/src/hello.txt" >/dev/null
$SUDO chmod 0777 "$W/src/pub"; $SUDO chmod 0755 "$W" "$W/src" "$W/mnt" "$W/copy"
if [ "${PROBE_INSTALL:-0}" = 1 ]; then
if command -v apt-get >/dev/null 2>&1; then
T 110 $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y -qq fuse3 fuse-overlayfs util-linux >/dev/null 2>&1 \
|| { T 110 $SUDO env DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1; \
T 110 $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y -qq fuse3 fuse-overlayfs util-linux >/dev/null 2>&1; }
elif command -v apk >/dev/null 2>&1; then
T 110 $SUDO apk add --no-cache fuse3 fuse-overlayfs util-linux setpriv shadow >/dev/null 2>&1
fi
fi
if ! command -v fuse-overlayfs >/dev/null 2>&1; then say route_a "skipped: no fuse-overlayfs"; exit 0; fi
# the command identity: a plain nonroot user, like LayerFS's CommandIdentity
if ! id lfsprobe >/dev/null 2>&1; then
if command -v useradd >/dev/null 2>&1; then $SUDO useradd -u 1500 -U -M -s /bin/sh lfsprobe 2>/dev/null
else $SUDO addgroup -g 1500 lfsprobe 2>/dev/null; $SUDO adduser -D -H -u 1500 -G lfsprobe -s /bin/sh lfsprobe 2>/dev/null; fi
fi
say cmd_user "$(id lfsprobe 2>/dev/null || echo absent)"
# run a shell snippet as the command identity, dropping every capability and
# setting no-new-privileges, as LayerFS's Docker Exec does
if command -v setpriv >/dev/null 2>&1; then
AS() { T 20 $SUDO setpriv --reuid=1500 --regid=1500 --clear-groups --inh-caps=-all --bounding-set=-all --no-new-privs sh -c "$1"; }
say cmd_launcher "setpriv"
else
AS() { T 20 $SUDO su -s /bin/sh lfsprobe -c "$1"; }
say cmd_launcher "su (no no-new-privs, no bounding-set drop)"
fi
# 1. the daemon side: root mounts with allow_other
if T 30 $SUDO fuse-overlayfs -o "allow_other,lowerdir=$W/src,upperdir=$W/up,workdir=$W/wk" "$W/mnt" 2>"$E/mount.err"; then
say daemon_mount_allow_other "ok $(awk -v m="$W/mnt" '$2==m{print $3" "$4}' /proc/mounts)"
else
say daemon_mount_allow_other "failed: $(head -c 200 "$E/mount.err" | tr '\n' ' ')"
say done ok; exit 0
fi
# 2. the command side, as uid 1500 with no capabilities
say cmd_caps "$(AS 'awk "/^CapEff|^CapBnd|^NoNewPrivs|^Seccomp:/{printf \"%s%s \", \$1, \$2}" /proc/self/status' 2>&1)"
say cmd_read "$(AS "cat $W/mnt/hello.txt" 2>&1 | head -c 120)"
say cmd_write "$(AS "echo from-cmd > $W/mnt/pub/c.txt && cat $W/mnt/pub/c.txt" 2>&1 | head -c 120)"
say cmd_write_owner "$(stat -c '%u:%g' "$W/mnt/pub/c.txt" 2>/dev/null || echo none)"
# must FAIL for parity: the command cannot unmount the Workspace
if AS "umount $W/mnt 2>/dev/null || fusermount3 -u $W/mnt 2>/dev/null || fusermount -u $W/mnt" >/dev/null 2>&1; then
say cmd_unmount "SUCCEEDED (parity broken)"
T 30 $SUDO fuse-overlayfs -o "allow_other,lowerdir=$W/src,upperdir=$W/up,workdir=$W/wk" "$W/mnt" 2>/dev/null
else say cmd_unmount "denied (parity ok)"; fi
# must FAIL for parity: no user namespace, so no private mount namespace
if AS "unshare -U true" >/dev/null 2>&1; then
say cmd_userns "allowed (parity broken: needs a seccomp/sysctl block)"
if AS "unshare -Urm sh -c 'mount --bind $W/mnt $W/copy && cat $W/copy/hello.txt'" >"$E/copy.out" 2>&1; then
say cmd_copy_mount "SUCCEEDED $(head -c 60 "$E/copy.out" | tr '\n' ' ') (a command can hold its own copy of the mount)"
else say cmd_copy_mount "failed: $(head -c 120 "$E/copy.out" | tr '\n' ' ')"; fi
else say cmd_userns "denied (parity ok)"; fi
say userns_sysctl "max_user_namespaces=$(cat /proc/sys/user/max_user_namespaces 2>/dev/null || echo ?) unprivileged_userns_clone=$(cat /proc/sys/kernel/unprivileged_userns_clone 2>/dev/null || echo n/a) apparmor_restrict=$(cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns 2>/dev/null || echo n/a)"
# can root here deny user namespaces for commands without Docker? (sysctl)
say root_can_block_userns "$([ -w /proc/sys/user/max_user_namespaces ] || $SUDO sh -c '[ -w /proc/sys/user/max_user_namespaces ]' && echo yes-sysctl-writable || echo no)"
# 3. a process left behind by a command is reaped (LayerFS uses an init as PID 1)
say pid1 "$(cat /proc/1/comm 2>/dev/null)"
# 4. teardown: the daemon side can unmount
if T 20 $SUDO fusermount3 -u "$W/mnt" 2>/dev/null || T 20 $SUDO umount "$W/mnt" 2>/dev/null; then say daemon_unmount ok; else say daemon_unmount failed; fi
$SUDO rm -rf "$W"; rm -rf "$E"
say done okStore probe (lfs-store-probe.sh)
#!/bin/sh
# LayerFS Store-volume probe: could a LayerFS Store live in DIR?
# LayerFS accepts a Store only on ext4, xfs or btrfs, checked with statfs
# (core/crates/layerfs-store-service/src/kernel_guard.rs), and uses SQLite in
# WAL mode with several processes, file locks, renames and in-place writes.
#
# Usage: sh lfs-store-probe.sh DIR
# PROBE_LOOP=1 instead of testing DIR itself, make a sparse ext4 image file
# DIR/lfs-store.img (PROBE_LOOP_MB, default 512), loop-mount it
# at /tmp/lfsstore-loop and test that (needs root or sudo).
# PROBE_KEEP=1 leave the loop mount in place (to test it again after a restart)
# Prints KEY=VALUE lines. One attempt per step, each with a timeout.
set -u
say() { printf '%s=%s\n' "$1" "$2"; }
T() { if command -v timeout >/dev/null 2>&1; then timeout -k 5 "$@"; else shift; "$@"; fi; }
D=${1:?usage: lfs-store-probe.sh DIR}
SUDO=""
if [ "$(id -u)" != 0 ] && sudo -n true 2>/dev/null; then SUDO="sudo -n"; fi
say store_probe_version 1
fsinfo() { # path -> type and magic as LayerFS's guard sees them
m=$(stat -f -c %t "$1" 2>/dev/null)
case "$m" in ef53) g=ext4;; 58465342) g=xfs;; 9123683e) g=btrfs;; *) g="rejected";; esac
echo "magic=0x$m guard=$g mount=$(awk -v p="$1" 'index(p,$2)==1 && length($2)>=n {n=length($2); l=$1" "$3} END{print l}' /proc/mounts)"
}
if [ "${PROBE_LOOP:-0}" = 1 ]; then
say dir_fs "$(fsinfo "$D")"
IMG="$D/lfs-store.img"; MNT=/tmp/lfsstore-loop
$SUDO mkdir -p "$MNT"
if ! grep -qs " $MNT " /proc/mounts; then
if [ ! -e "$IMG" ]; then
T 30 $SUDO truncate -s "${PROBE_LOOP_MB:-512}M" "$IMG" && T 60 $SUDO mkfs.ext4 -q -F "$IMG" >/dev/null 2>&1 \
&& say loop_image "created $IMG" || { say loop_image "failed to create (truncate/mkfs.ext4)"; exit 0; }
else say loop_image "reused existing $IMG"; fi
if T 30 $SUDO mount -o loop "$IMG" "$MNT" 2>/tmp/lfsstore-mount.err; then say loop_mount ok
else say loop_mount "failed: $(head -c 200 /tmp/lfsstore-mount.err | tr '\n' ' ')"; exit 0; fi
else say loop_mount "already mounted"; fi
$SUDO chmod 1777 "$MNT"
say loop_dev "$(awk -v m="$MNT" '$2==m{print $1}' /proc/mounts)"
D="$MNT"
fi
say store_fs "$(fsinfo "$D")"
P="$D/lfsstoreprobe.$$"
mkdir -p "$P" || { say store_write "failed: cannot mkdir in $D"; exit 0; }
[ -f "$D/marker" ] && say marker_before "$(cat "$D/marker")" || say marker_before none
date -u +%Y-%m-%dT%H:%M:%SZ > "$D/marker"
# in-place write at an offset, then read back
if command -v python3 >/dev/null 2>&1; then
say pwrite "$(T 20 python3 -c "
import os,sys
p=sys.argv[1]+'/f'; fd=os.open(p,os.O_RDWR|os.O_CREAT,0o600); os.pwrite(fd,b'A'*8192,0); os.pwrite(fd,b'B',4096); os.fsync(fd)
print('ok' if os.pread(fd,1,4096)==b'B' else 'mismatch')" "$P" 2>&1 | tail -1)"
# rename over an existing file
say rename_over "$(T 20 python3 -c "
import os,sys
d=sys.argv[1]; open(d+'/a','w').write('new'); open(d+'/b','w').write('old'); os.rename(d+'/a',d+'/b'); print('ok' if open(d+'/b').read()=='new' else 'mismatch')" "$P" 2>&1 | tail -1)"
# SQLite WAL with a writer process and a separate reader process
say sqlite_wal "$(T 60 python3 -c "
import sqlite3,sys,subprocess
db=sys.argv[1]+'/s.sqlite'
c=sqlite3.connect(db); m=c.execute('pragma journal_mode=wal').fetchone()[0]
c.execute('create table t(x)'); c.execute('insert into t values (1)'); c.commit()
c.execute('begin immediate'); c.execute('insert into t values (2)')
code='import sqlite3,sys; r=sqlite3.connect(sys.argv[1],timeout=0); print(r.execute(\"select count(*) from t\").fetchone()[0]);\ntry:\n r.execute(\"begin immediate\"); print(\"second-writer-allowed\")\nexcept sqlite3.OperationalError as e: print(\"second-writer-refused:\"+str(e))'
out=subprocess.run([sys.executable,'-c',code,db],capture_output=True,text=True,timeout=20)
c.commit()
w=out.stdout.split()
print(('mode='+m+' reader_saw='+w[0]+' second_writer='+('refused (ok)' if 'refused' in out.stdout else 'ALLOWED')) if w else 'reader-failed:'+out.stderr[-120:])" "$P" 2>&1 | tail -1 | tr '\n' ' ')"
else
say pwrite "skipped: no python3"; say rename_over "skipped: no python3"; say sqlite_wal "skipped: no python3"
fi
# advisory locks across processes (LayerFS also takes a Linux directory lock)
if command -v flock >/dev/null 2>&1; then
( flock -x 9; sleep 3 ) 9>"$P/lock" &
sleep 1
if T 5 flock -n -x "$P/lock" true 2>/dev/null; then say flock "NOT exclusive (second lock granted)"; else say flock "exclusive ok"; fi
wait
else say flock "skipped: no flock"; fi
rm -rf "$P"
if [ "${PROBE_LOOP:-0}" = 1 ] && [ "${PROBE_KEEP:-0}" != 1 ]; then
T 20 $SUDO umount /tmp/lfsstore-loop && say loop_unmount ok || say loop_unmount failed
fi
say done okL1 probe (lfs-sandbox-probe.sh, version 3: timeout -k so it can stop docker run)
#!/bin/sh
# LayerFS sandbox probe: can this environment host LayerFS's FUSE mounts?
# Prints KEY=VALUE lines. Writes only under /tmp/lfsprobe (and, with
# PROBE_INSTALL=1, installs fuse3 + fuse-overlayfs through the package manager).
#
# Levels:
# L1 inside this environment: /dev/fuse, capabilities, privilege to mount,
# a real FUSE mount (fuse-overlayfs) as root and as a plain user.
# L2 nested Docker with LayerFS's exact Sandbox flags: /dev/fuse device,
# CAP_SYS_ADMIN, apparmor=unconfined, no-new-privileges and LayerFS's
# seccomp filter (no user namespaces); a FUSE mount inside it; and the
# filesystem a Docker volume gets (the Store needs ext4, xfs or btrfs).
#
# Every step is one attempt with its own timeout; a failure is reported, not
# retried. Usage: sh lfs-sandbox-probe.sh (env PROBE_INSTALL=1 to apt/apk
# install fuse3 and fuse-overlayfs if missing; PROBE_DOCKER=0 to skip L2).
set -u
W=/tmp/lfsprobe
mkdir -p "$W/src" "$W/mnt" "$W/mnt2" "$W/up" "$W/wk"
echo "probe-file" > "$W/src/hello.txt"
say() { printf '%s=%s\n' "$1" "$2"; }
T() { if command -v timeout >/dev/null 2>&1; then timeout -k 5 "$@"; else shift; "$@"; fi; }
SUDO=""
if [ "$(id -u)" != 0 ]; then
if sudo -n true 2>/dev/null; then SUDO="sudo -n"; fi
fi
say probe_version 3
say uid "$(id -u)"
say root_route "$([ "$(id -u)" = 0 ] && echo root || { [ -n "$SUDO" ] && echo sudo || echo none; })"
say kernel "$(uname -srm)"
say os "$(. /etc/os-release 2>/dev/null; echo "${PRETTY_NAME:-unknown}")"
say virt "$(systemd-detect-virt 2>/dev/null || echo unknown)"
# --- L1: device, module, capabilities -------------------------------------
if [ -c /dev/fuse ]; then say dev_fuse "$(ls -l /dev/fuse | awk '{print $1" "$3":"$4}')"; else say dev_fuse absent; fi
say fuse_in_kernel "$(grep -qw fuse /proc/filesystems && echo yes || echo no)"
say dev_kvm "$([ -c /dev/kvm ] && echo present || echo absent)"
capbit() { v=$(printf '%d' "0x$1" 2>/dev/null || echo 0); echo $(( (v >> 21) & 1 )); }
ce=$(awk '/^CapEff/{print $2}' /proc/self/status)
say cap_eff "$ce sys_admin=$(capbit "$ce")"
if [ -n "$SUDO" ]; then
cr=$($SUDO awk '/^CapEff/{print $2}' /proc/self/status)
say cap_eff_root "$cr sys_admin=$(capbit "$cr")"
fi
say no_new_privs "$(awk '/^NoNewPrivs/{print $2}' /proc/self/status)"
say seccomp_mode "$(awk '/^Seccomp:/{print $2}' /proc/self/status)"
say userns "$(T 10 unshare -U true 2>/dev/null && echo allowed || echo denied)"
fstype() { awk -v p="$1" 'index(p, $2) == 1 && (p == $2 || substr(p, length($2) + 1, 1) == "/" || $2 == "/") { if (length($2) >= n) { n = length($2); t = $3 } } END { print t }' /proc/mounts; }
say fs_root "$(fstype /)"
say fs_tmp "$(fstype /tmp)"
fm=$(command -v fusermount3 || command -v fusermount || true)
if [ -n "$fm" ]; then say fusermount "$fm mode=$(stat -c %A "$(readlink -f "$fm")")"; else say fusermount absent; fi
say user_allow_other "$(grep -qs '^user_allow_other' /etc/fuse.conf && echo yes || echo no)"
# privilege to mount at all (tmpfs needs CAP_SYS_ADMIN)
if [ "$(id -u)" = 0 ] || [ -n "$SUDO" ]; then
if T 20 $SUDO mount -t tmpfs lfsprobe "$W/mnt2" 2>"$W/tmpfs.err"; then
say mount_tmpfs ok; T 20 $SUDO umount "$W/mnt2"
else say mount_tmpfs "failed: $(head -c 160 "$W/tmpfs.err" | tr '\n' ' ')"; fi
else say mount_tmpfs "skipped: no root"; fi
# optional install of a FUSE test program
if ! command -v fuse-overlayfs >/dev/null 2>&1 && [ "${PROBE_INSTALL:-0}" = 1 ]; then
if command -v apt-get >/dev/null 2>&1; then
T 110 $SUDO env DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1
T 110 $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y -qq fuse3 fuse-overlayfs >/dev/null 2>&1
elif command -v apk >/dev/null 2>&1; then
T 110 $SUDO apk add --no-cache fuse3 fuse-overlayfs >/dev/null 2>&1
fi
fi
say fuse_overlayfs "$(command -v fuse-overlayfs || echo absent)"
fuse_try() { # $1 label, $2.. command prefix
label=$1; shift
rm -rf "$W/up" "$W/wk"; mkdir -p "$W/up" "$W/wk"
if T 30 "$@" fuse-overlayfs -o "lowerdir=$W/src,upperdir=$W/up,workdir=$W/wk" "$W/mnt" 2>"$W/$label.err"; then
got=$(T 10 cat "$W/mnt/hello.txt" 2>/dev/null)
typ=$(awk -v m="$W/mnt" '$2==m{print $3}' /proc/mounts)
say "$label" "ok type=$typ read=$got"
T 20 $SUDO fusermount3 -u "$W/mnt" 2>/dev/null || T 20 $SUDO umount "$W/mnt" 2>/dev/null
else
say "$label" "failed: $(head -c 200 "$W/$label.err" | tr '\n' ' ')"
fi
}
if command -v fuse-overlayfs >/dev/null 2>&1; then
if [ "$(id -u)" = 0 ] || [ -n "$SUDO" ]; then fuse_try fuse_mount_root $SUDO; else say fuse_mount_root "skipped: no root"; fi
if [ "$(id -u)" != 0 ]; then fuse_try fuse_mount_user env; fi
else
say fuse_mount_root "skipped: no fuse-overlayfs"
fi
# --- L2: nested Docker with LayerFS's Sandbox flags ------------------------
if [ "${PROBE_DOCKER:-1}" = 1 ]; then
if command -v docker >/dev/null 2>&1 && T 20 $SUDO docker info >/dev/null 2>&1; then
say docker "$($SUDO docker info --format '{{.ServerVersion}} runtime={{.DefaultRuntime}} storage={{.Driver}} cgroup={{.CgroupVersion}}' 2>/dev/null)"
cat > "$W/seccomp.json" <<'EOF'
{"defaultAction":"SCMP_ACT_ALLOW","syscalls":[{"names":["unshare","clone"],"action":"SCMP_ACT_ERRNO","errnoRet":1,"args":[{"index":0,"value":268435456,"valueTwo":268435456,"op":"SCMP_CMP_MASKED_EQ"}]},{"names":["clone3"],"action":"SCMP_ACT_ERRNO","errnoRet":38}]}
EOF
if T 110 $SUDO docker run --rm --pull=missing \
--device /dev/fuse --cap-add SYS_ADMIN \
--security-opt apparmor=unconfined --security-opt no-new-privileges=true \
--security-opt seccomp="$W/seccomp.json" \
alpine:3.20 sh -c 'apk add --no-cache -q fuse3 fuse-overlayfs >/dev/null 2>&1; mkdir -p /s /m /u /w; echo nested > /s/f; fuse-overlayfs -o lowerdir=/s,upperdir=/u,workdir=/w /m && cat /m/f && grep " /m " /proc/mounts | cut -d" " -f3; unshare -U true 2>/dev/null && echo userns=allowed || echo userns=denied' \
>"$W/l2.out" 2>"$W/l2.err"; then
say l2_fuse_in_container "ok $(tr '\n' ' ' < "$W/l2.out")"
else
say l2_fuse_in_container "failed: $(tail -c 240 "$W/l2.err" | tr '\n' ' ')"
fi
if T 60 $SUDO docker run --rm -v lfsprobe-vol:/v alpine:3.20 stat -f -c %T /v >"$W/vol.out" 2>&1; then
say l2_volume_fs "$(cat "$W/vol.out")"
else say l2_volume_fs "failed: $(head -c 160 "$W/vol.out")"; fi
T 30 $SUDO docker volume rm lfsprobe-vol >/dev/null 2>&1
else
say docker "absent or not running"
fi
fi
say done ok
Activity
[-]Sandbox experiment: can LayerFS mount in agent sandboxes? Local stand-ins first[/-][+]Sandbox backends: which agent sandboxes can replace LayerFS's Docker container? Local stand-ins first[/+]yifanxuaaa commentedon Oct 30, 2026
Microsandbox (libkrun microVM): every level passed, and route A works except for one design change
msb 0.7.11 (official release tarball, checksum verified), Debian 13 guest, kernel 6.12.113 aarch64. Stands in for E2B, Vercel Sandbox and Modal's VM runtime.
/dev/fuseis present; a realfuse-overlayfsmount works; PID 1 (init.krun) reaps orphansallow_other; a uid-1500 command with no capabilities reads and writes but cannot unmount. It can create a user namespace and copy the mount. Root closes that withuser.max_user_namespaces=0(thenunsharefails with ENOSPC); the daemon needs no user namespacescalls_real.mjs inner workspace15/15 with Docker 29.9.2 inside the guest. Route A can't run end to end until a microVM backend existsBackend contract
--copy-filekeeps 0600;msb copy)msb logs --job)-p 127.0.0.1:H:Gbinds host loopback onlyls --label)The design change (B2): two VMs never share a kernel or a Store disk. So with route A, a project's Sandboxes become several daemons inside one VM per project (or per Store), not one VM each. L4 showed the SDK and several daemons sharing one guest kernel.
Open:
msb exec -udrops capabilities and sets no-new-privileges by itself (the probe usedsetpriv, so plan on the backend wrapping commands);Found along the way:
docker.io26.1.5 (Engine API 1.45) is older than the 1.48 LayerFS needs./var/lib/dockeron a real ext4.timeoutneeds-kso it can stopdocker run.A shared host folder (virtio-fs) shows up as
virtiofs, has attribute-cached coherence, and keeps guest ownership in an xattr, so it isn't a Store volume.Hosted providers' kernels, policies and volumes are untested; each needs its own run. All test VMs and volumes were removed.
yifanxuaaa commentedon Oct 30, 2026
Daytona open source, run locally: FUSE works, but there's no volume that can hold the Store
Daytona OSS docker-compose from release v0.190.0. The published images report v0.187.0 (there's no 0.190.0 tag on Docker Hub). It ran locally on Docker Desktop (arm64), with no Daytona cloud account. It's a best case for Daytona's container sandboxes: the cloud may use a different runtime (Sysbox or VMs).
What a sandbox is here: a runc container with
Privileged=true, run by the runner's own Docker. It has onlylabel=disable(no seccomp, no AppArmor, no user namespace), and PID 1 is Daytona's daemon, which reaps children. The OSS runner accepts amd64 images only, so on Apple Silicon the sandbox ran under emulation./dev/fuseis present, root (via sudo) has every capability, andfuse-overlayfsmounts as root and as a plain userallow_othermount works; a uid-1500 command reads and writes but can't unmount; it can create a user namespace and copy the mount. Root can writeuser.max_user_namespaces, but a privileged sandbox shares the runner's kernel, so that would change every sandbox on the runnerBackend contract
@sha256snapshots (a bare:latestis refused)Verdict: a Daytona container sandbox can mount FUSE, but it can't be a route-A backend as it is: there's no local volume for the Store (B2), commands can't be given a separate uid (B7), and nothing blocks user namespaces per sandbox (B4). A Daytona VM sandbox class exists in the API but has no implementation in the open-source runner, so it needs a cloud account to test.
All of the run's containers, volumes, networks and pulled images were removed.
yifanxuaaa commentedon Oct 30, 2026
Apple Container (one VM per container): FUSE works with
--cap-add SYS_ADMIN; no shared Store, and the host endpoint is unconfirmedcontainer1.6.1 (Homebrew; Apple's CLI), Linux 6.18.35 aarch64 guest, Debian 12 image. It stands in for platforms that run one VM per container./dev/fuseis present but SYS_ADMIN isn't granted (Operation not permitted). With--cap-add SYS_ADMIN: PASS (fuse.fuse-overlayfsmount and read). No--deviceoption is neededallow_othermount works; uid 1500 can read and write but can't unmount; it can create a user namespace and copy the mount. With--cap-add SYS_RESOURCE, root can mount a fresh procfs and setuser.max_user_namespaces=0, after which the command'sunshare -Ufails. The VM belongs to one sandbox, so that VM-wide setting is acceptable. There's no--security-opt seccomp--cap-add NET_ADMINnot tried)Backend contract
--name, which refuses duplicatesstorage device attachment is invalid), and separate kernels couldn't share SQLite locks anyway. Several daemons in one VM wasn't testedcontainer cpworks only on a running container and drops modes (0755 and 0600 both became 0644). A pinned image carrying the daemon avoids it--initgives a PID 1 that reaps orphans-p 127.0.0.1:H:Gbinds host loopback, but the forwarder loggedNo route to host; the host couldn't reach any VM, while VM-to-VM TCP worked. Likely macOS Local Network privacy blocking the runtime helper; not confirmedsetprivwrapper); no way to kill one exec except by PID from another execls --format jsonon the clientVerdict: one VM per sandbox gives each its own kernel and its own Store disk, so route A would mean one container per project running several daemons, like the Microsandbox result. Apple Container is a local tool, not a hosted provider; it would matter only as a macOS backend, and then B6 must work first.
All the run's containers, volumes and images were removed, and the container service is stopped.
yifanxuaaa commentedon Oct 30, 2026
Apple Container, iteration 2: one VM per project works, and real LayerFS passes inside it
Iteration 1's blockers came from treating every Sandbox as its own VM. This iteration tested one Apple container (one VM) per project: it holds the project's Store on its named volume and runs several Workspaces, each in its own mount namespace.
/dev/vdc, which LayerFS's filesystem guard accepts. In-place writes, rename-over, SQLite WAL with a second process (the second writer is refused), and flock all pass. After the container was deleted and a new one started on the same volume, the earlier marker was still there. A loop-mounted ext4 on the root disk also passesallow_otherFUSE mount; neither is visible in the VM's own namespace or in the other. Commands sent in withcontainer exec -u 0 … nsenter -t <pid> -m setpriv --reuid=1500 … --bounding-set=-all --no-new-privsran at the same time as uid 1500, with no capabilities and no-new-privs. Each read its own mount and saw the other Workspace as empty. Unmount was denied.user.max_user_namespaces=0(needs--cap-add SYS_RESOURCE) blocks user namespaces VM-wide. This closes iteration 1's B7 capability gapcontainer exec -icarried 4 MiB to an echo server on the guest's 127.0.0.1:7000 and back, with matching sha256, and an HTTP request round-tripped. Opening a bridge took about 0.06 s. A fresh-p 127.0.0.1:H:Galso worked this time; iteration 1's failure was transient, cause unknowncontainer exec -i … tar -x -pkept 0755, 0600 and 0700 (the container must be running)calls_real.mjs inner workspace15 calls in 315 ms, first attempt. Flags--cap-add SYS_ADMIN --cap-add NET_ADMIN --init, Docker's official static 29.9.2. The sealed bundle and theafa1ef9c…image came in over a host bind mount, not the networkFound along the way:
/proc/sysremounted read-write.container stopfailed on the nested cgroups (errno 95 writingcgroup.kill). A backend should stop the inner dockerd before stopping the VM.Not proven yet:
layerfs-daemonitself running in a per-Workspace namespace with no Docker. The isolation test usedfuse-overlayfs, and L4 went through Docker. That needs a route-A backend, or a stand-in, to drive the daemon.-p.Updated contract for this design:
All containers, volumes and images were removed, and the container service is stopped.
yifanxuaaa commentedon Oct 30, 2026
Lima VM (EC2-style) and a real Firecracker microVM: both pass
Two iterations. In the first, Lima 1.0.4 couldn't start a vz VM on this host: its network stack, gvisor-tap-vsock v0.8.2, watches the folder that holds
/etc/resolv.confand fails on a socket symlink there. Lima v2.2.1 (official release, checksum verified, run from a scratch folder) bundles v0.8.9, which drops that watcher. With it, vz with nested virtualization boots, and the VM has/dev/kvm.Plain Linux VM: Ubuntu 24.04 arm64, kernel 6.8, stands in for EC2
allow_otherok; the uid-1500 command reads and writes; unmount denied. User namespaces are allowed, but copying the mount was blocked by Ubuntu's AppArmor userns restriction.user.max_user_namespaces=0denies them outright, VM-widesandbox_real(16 FUSE Workspaces in one Sandbox, Exec) ok in 1095 ms;calls_real workspace15 calls in 488 ms;runtime_realok. The Store volume is ext4Firecracker v1.17.0 guest, nested in that VM, stands in for E2B and Vercel
The guest booted Firecracker's CI kernel 6.18.56 and its CI Ubuntu rootfs, with a second ext4 drive as the Store and vsock (no network device).
CONFIG_FUSE_FS=y, plus OVERLAY, EXT4, XFS, USER_NS, SECCOMP_FILTER and VSOCK. They don't set BTRFS, VIRTIO_FS or AppArmor. Providers build their own kernels, so FUSE has to be checked on each one./dev/fuse: present at boot via devtmpfs, so nomknodis needed. L1 PASS./dev/vdb, found by its label, ext4, read-write.--bridge=none --iptables=false(the guest has no network), so Docker networking wasn't tested.CONNECT <port>; guest to host connects to CID 2. The daemon's TCP endpoint would need a small vsock relay on one side.Contract, for a VM or microVM per project
setprivplus a seccomp filter or the sysctl.Not shown:
Everything was deleted: the VM, the Firecracker files, the downloaded Lima and images. Homebrew's Lima and Colima weren't touched.
yifanxuaaa commentedon Oct 30, 2026
Daytona open source, iteration 2: the Store on a loop ext4 works, and real LayerFS passes in a sandbox
Iteration 1's blockers were the S3 volumes and the amd64-only runner. This iteration got around both.
docker commit, not paused) keeps the image file, but it's only crash-consistent while mounted, and it loses sparsenessnsenter, reads and writes but can't unmount. User namespaces stay open unless a seccomp filter is added: the sysctl would affect the whole runner kernelapps/runner/pkg/docker. The patch reads the architecture from an env var. Daytona's own in-sandbox daemon is still shipped amd64 only, so it ran under emulation here; a real arm64 host would need an arm64 build of itcalls_real.mjs inner workspace15 calls in 394 ms in a native arm64 sandbox, with Docker's official static 29.9.2 (Engine API 1.56),/var/lib/dockeron a loop ext4, the sealed bundle and theafa1ef9c…image-L 127.0.0.1:H:127.0.0.1:Pcarried raw TCP to a listener in the sandbox, bound to the host's loopbacksudo setpriv …gives uid 1500, all capabilities 0, NoNewPrivs 1 and the env. Commands still go through bash. No stdin EOF for async execs: Daytona's daemon holds the input open until the command ends, and appends a newline. Workaround: upload input as a file and redirect from itFound along the way:
Verdict: Daytona's container sandboxes can work in a one-sandbox-per-project design:
setpriv;Gaps for a backend:
This is still the local open-source stack only.
Everything the run created was removed: the sandboxes, loop devices, the compose stack and its volumes, and the pulled and built images.
yifanxuaaa commentedon Oct 30, 2026
Route A end to end: real LayerFS ran with no Docker at all
Result: the unmodified sealed bundle (
build-linux-bb5f636…, source7ab54bcf1a, hashes match its manifest) passedcalls_real.mjs inner workspace, 15 of 15 calls. It ran inside an Apple Container VM (Linux 6.18.35 aarch64) with no Docker in it:docker,dockerd,containerd,runc,crunandtiniwere all absent.user.max_user_namespaces=0;Each run used a fresh ext4 Store and the Disposable profile.
How: a test-only stand-in (about 450 lines of Node, no dependencies, never committed) answers the Engine API v1.54 calls LayerFS makes on
/var/run/docker.sock:/layerfs-store/dev/fuseand basic devices are in/devnsenterinto the namespaces, thensetprivto the requested uid:gid with no capabilities and no-new-privs; stdin, stdout and stderr go over the hijacked streamHonest limits of the stand-in: inspect echoes the Config and HostConfig fields LayerFS verifies (including the seccomp JSON) from the create request; it doesn't enforce them all. It does enforce:
/devdevice set;It does not enforce seccomp (the VM-wide sysctl stands in for it), a device cgroup, or any cgroup limits.
Isolation checks (two Sandboxes, real LayerFS Workspace mounts, through
workspace.exec):unshare -Uworked and a command could copy the mount. After it, both failed (No space left on device).VM requirements found:
--cap-add NET_ADMIN, to bring up loopback in a new network namespace;--cap-add SYS_RESOURCE, for the sysctl;/proc/<pid>/task/<pid>/children, so PID 1 is found by scanning/proc.What this proves: LayerFS's SDK and daemon need nothing Docker-specific. Any Linux VM that gives root,
/dev/fuse, namespaces and an ext4 disk can host the Sandbox contract. The backend's job reduces to the stand-in's job.What a real route-A backend still needs:
nsenter(or this kind of agent inside the VM);-pon 127.0.0.1), Daytona (SSH gateway), Apple (exec stdio) and Firecracker (vsock) showed;Both VMs, the image and all the tars were removed, and the container service is stopped.
yifanxuaaa commentedon Oct 30, 2026
Route A in a Microsandbox microVM: real LayerFS with no Docker, and the Store survives the VM
This repeats the no-Docker run in a libkrun microVM, which stands in for E2B, Vercel and Modal's VM runtime, using the test-only Engine API stand-in from the route-A comment. It used the unmodified sealed bundle (
build-linux-bb5f636…) on msb 0.7.11, with Linux 6.12.113, 4 GiB and 4 vCPUs. Everything went in over virtio-fs shares, with no guest network. In the guest,docker,dockerd,containerd,runc,crunandtiniwere all absent, andpsshowed only/init.krunand the stand-in.calls_real.mjs inner workspace, user namespaces openuser.max_user_namespaces=0sandbox_real.mjs inner: 16 Workspaces in one Sandbox, sysctl setworkspace.execumountandumount -lare denied. Each Sandbox has its own mount, network and PID namespace and sees only its own FUSE mount; the VM itself sees none. A leftover process is reaped by the Sandbox's PID 1. Under the sysctl,unshare -Uand copying the mount fail; without it they succeed (the known gap)msb stopandmsb rm, a new microVM with only the Store disk attached found the Store unchanged (same sha256) andintegrity_checkok.LayerFS.openlisted branchesa,bandmain, and read back a file that an exec in the earlier VM wrote and committedLayout: two
diskvolumes, both ext4 block devices in the guest:Compared with Apple Container:
--cap-add SYS_ADMIN,NET_ADMINandSYS_RESOURCE./proc/<pid>/task/<pid>/childrenfile, which Apple's lacks.sysctl -wworks on Microsandbox; Apple needed a fresh procfs mount.Timings are single samples, not measurements.
Everything was removed: both microVMs, both volumes, the image and the share tars.
yifanxuaaa commentedon Oct 30, 2026
Real LayerFS inside a Firecracker microVM: passes with no Docker
This closes the Firecracker gap from the Lima and Firecracker comment, where only the probes had run. It used the unmodified sealed bundle
build-linux-bb5f636…, all 192 hashes matching its manifest, and the test-only Engine API stand-in from the route-A comment.Rig: Firecracker v1.17.0 with the CI kernel 6.18.56 (4 vCPUs, 4096 MiB, no network), nested in a Lima 2.2.1 vz VM. The guest root was the Sandbox image's own root filesystem (Debian 12, Node 24), unpacked from
sha256:afa1ef9c…. A second virtio-blk ext4 drive held the Store and the Sandbox data. A small guest init ran the stand-in and the test.calls_real.mjs inner workspace, user namespaces openuser.max_user_namespaces=0unshare -Uas the command identity denieddockerd,containerd, shim,runc,crunordockerprocess or binaryNested-rig note: with ordinary 4 KiB guest memory, the guest was very slow. Kernel-to-init took 28–55 s and
node -e 0took 20 s, so the first attempt timed out after 3 calls. The cause was first-touch page faults passing through two hypervisor levels. Backing guest memory with 2 MiB huge pages (Firecracker"huge_pages": "2M") cut boot to 1.9 s andnode -e 0to 0.4 s; boot, both runs and poweroff took 8.2 s in all. This is a cost of nesting Firecracker on a Mac, not of LayerFS or Firecracker on bare metal.Not done:
NF_TABLES_INET,DUMMY,MACVLANandIPVLAN. Port publishing is untested, and no blocker was found.Provenance caveat: the stand-in source was being revised by the parallel Microsandbox run when this run copied it, so which of its two versions ran here isn't recorded. Both versions passed
calls_realelsewhere.Timings are single samples, not measurements. Everything was deleted: the VMs, drives, downloads and the short Lima home.
Where this leaves #512
Real LayerFS has now passed with no Docker in three VM kinds:
It also passed with Docker inside on Microsandbox, Apple Container, local Daytona and a Lima VM. The proposed backend is in draft PR #515.
Merge pull request #526 from jiangkoumo/vm515-v2 into layerfs_v2_0_0
Mola-maker commentedon Nov 1, 2026
感谢提出这个 issue，也感谢 AI 协作工具在整个排查过程中做的分析 🙏
对 AI agent 的使用来说，这个问题的解决会直接改善 agent 能否用现成 sandbox 标准替代 Docker，决定 agent 环境的部署成本。
两点建议：