Repository navigation
Sandbox backends: twelve-cell speed comparison, Docker vs the no-Docker VM route (sub-issue of #512) #516
Description
Goal
Compare LayerFS's speed on the no-Docker VM route (#512, design in PR #515) with today's Docker route, using the twelve R7 cells (C01–C12) as the workload. Same bundle, same Sandbox image, same cell runner, and the same stand-in on every VM; only the platform changes.
Candidates
| # | Candidate | Route |
|---|---|---|
| 0 | Docker Desktop | today's Docker backend (baseline) |
| 1 | Microsandbox microVM (libkrun) | no Docker: Sandboxes as namespaces through the #512 test-only Engine API stand-in |
| 2 | Apple Container VM | no Docker, same stand-in |
| 3 | Lima vz VM (Ubuntu) | no Docker, same stand-in |
Not timed:
- Firecracker: here it can only run nested inside a Lima VM, where page faults cross two hypervisors, so its timings would describe the nesting. It stays a pass/fail check.
- Daytona open source: its sandboxes are privileged containers on Docker Desktop, so it would re-measure Docker Desktop.
- Hosted providers: they need accounts and run on other hardware.
The twelve cells
The bodies are exactly those in benchmark/cluster-two/cells/workloads.py at the parent of 196c714b0b, the commit that removed the harness on 2026-10-20. The last baseline is TIMING-BASELINE-20261010.md.
| Cell | Command | Base |
|---|---|---|
| C01 | create 1,000 files | empty |
| C02 | create 1,000, stat 1,000 | empty |
| C03 | create 1,000, remove them | empty |
| C04 | 100 directories, 1,000 files | empty |
| C05 | that tree, then find |
empty |
| C06 | write 64 MiB | empty |
| C07 | write 64 MiB, copy it | empty |
| C08 | write 64 MiB, read it | empty |
| C09 | read a 64 MiB base file | committed 64 MiB file |
| C10 | copy a 64 MiB base file | committed 64 MiB file |
| C11 | overwrite a 64 MiB base file | committed 64 MiB file |
| C12 | git init, 100 files, add, commit |
empty |
Method
-
A new runner, not the old one. The old R7 runner is pinned to the 2026-10-09 source, its own runtime binary and Docker CLI calls, so reviving it would measure old code. A small runner runs the same twelve bodies through the current SDK (
workspace.exec). Its numbers compare across the four candidates here, not with the 2026-10-10 baseline. -
Cache state, class B as before:
- per cell, a fresh project and Store;
- an untimed setup (the committed base for C09–C11);
- one untimed identical run on an earlier mount, then its terminal unmount;
- then a fresh mount for the timed run.
Every phase pays for its own work, with no warm-cache credit beyond that declared state.
-
One sample per cell per candidate. No reruns, no best-of. Failed and ineligible cells are kept.
-
What is recorded per cell:
- the command time on the Sandbox's own clock, and the
workspace.execwall on the SDK side; - mount, unmount and Commit times;
- the verifier's result (files, sizes and contents as the cell expects);
- the daemon's request and statement counts where the current telemetry exposes them;
- the Store and overlay bytes.
- the command time on the Sandbox's own clock, and the
-
Pinned identities:
- the sealed bundle
build-linux-bb5f636…(source7ab54bcf1a, hashes checked against its manifest); - Sandbox image
sha256:afa1ef9c…; - the runner's and the stand-in's sha256;
- each VM's kernel, vCPUs and memory;
- Disposable/WAL/OFF;
LAYERFS_CONSTRUCTION_WORKERS=1.
- the sealed bundle
-
Interference: sampling commands on this Mac take one host-wide lock, so no two candidates sample at once. The 1-minute load is recorded with every sample. Preparation on other candidates may overlap and is declared. The VMs get 4 vCPUs and 4 GiB each, so that three VMs and Docker Desktop fit in this Mac's 36 GB without swapping; Docker Desktop keeps its own 8 CPUs and 8 GB and is shared with idle containers, which is recorded.
Rules
- Each command is bounded at 2 minutes, and each test invocation at 120 s.
- One attempt per step.
- Disposable Stores only, never the owner's.
- Every VM, volume and image the run creates is removed afterwards.
Results
- 0 Docker Desktop
- 1 Microsandbox
- 2 Apple Container
- 3 Lima
- Comparison table, with the exact counts first and timings second
Activity
yifanxuaaa commentedon Oct 30, 2026
Twelve-cell comparison: the no-Docker VM route is as fast as Docker Desktop or faster, and each exec call costs ~4 ms instead of ~42 ms
All four candidates ran all twelve cells: 48 of 48 samples PASS their verifier, one sample each, no reruns. All samples used:
cells.mjsf68e67b9…);build-linux-bb5f636…, hashes matching its manifest);sha256:afa1ef9c…;704bfa37…) on every VM;LAYERFS_CONSTRUCTION_WORKERS=1.A host-wide lock ran one sample at a time: 10 of 13 recorded waits were 0 s, the longest 4 s, and the host's 1-minute load was 1.9–3.1.
1. Same work everywhere (counts first)
These counts are equal across all four candidates in every cell:
UpToDate.The exceptions are small and explained:
So the timings below compare the same filesystem work.
2. Command time on the Sandbox's own clock (ms; ratio to Docker Desktop)
findgit init, 100 files, add, commit3. The other phases (geometric mean of the ratio to Docker Desktop, over twelve cells)
workspace.execwall (SDK side)Fixed overhead per exec (the SDK's exec wall minus the command's own time; median over twelve cells):
That is the Docker Engine's exec create, attach and inspect against the stand-in's direct namespace launch. It's paid once per tool call, so it matters most for LayerFS's common per-tool-call case.
4. How to read it
host.docker.internal, with 8 CPUs, against 4 for the VMs. Part of its mount and exec cost belongs to Docker Desktop's setup, not to Docker itself.Conclusion: running LayerFS with no Docker, one VM per project and Sandboxes as namespaces, costs nothing in filesystem speed here. It makes each exec call about 38 ms cheaper and mount about 4 ms cheaper. Commit is unchanged. Lima is about 7% slower on command time, probably because of its older kernel. Git (C12) is 12–23% slower in every VM, the only cell where all three VMs lose.
Suggested next step: Docker inside the same Lima VM as a fourth arm, giving the same kernel and the same CPUs with only the backend changed. That isolates the backend from Docker Desktop's own costs.
All VMs, volumes, images and helper containers the run made were removed.
yifanxuaaa commentedon Oct 30, 2026
Extra reference, not the baseline: the same cells with no LayerFS
The four candidates above all run LayerFS. This adds a reference run: plain Docker Desktop, with no LayerFS and no FUSE. It ran:
sha256:afa1ef9c…), environment and in-container clock (date +%s%N);setpriv, with no capabilities and no-new-privs;--network none), one sample per cell, under the same host lock (acquired after 0 s every time, load 2.15–2.30).There are two variants: a fresh Docker volume per cell (ext4 in Docker Desktop's VM), and the container's own overlayfs root. Every native cell left the expected end state, so all 24 pass.
Command time on the in-container clock (ms; the ratio is to native on a volume)
findgit init, 100 files, add, commitHow to read it
The cost is per FUSE request, not per platform. Every LayerFS candidate sits at roughly the same multiple of native. The extra time divided by the body's FUSE requests:
This matches the R7 records' ~60 µs per request. Removing Docker changes this by only ~10%. The per-request path through FUSE, the daemon and the Overlay is the cost, wherever it runs.
Process-heavy cells look better (C02, C04, C05, about 2×), because spawning ~1,000
stat,mkdirortouchprocesses takes about 300 ms even natively and dominates. Cells made of plain shell redirections or one large file show the full FUSE cost: 10–30×.Native doesn't do LayerFS's other work. It has no mount, no Commit, no history and no isolation from the base. The LayerFS columns are command time only. Mount (~6–17 ms), Commit (3–630 ms) and the per-exec overhead come on top, as in the comparison above.
Same caveats as before:
Where the candidates differ from the LayerFS Docker Desktop baseline
The same split over all twelve cells, from the warm-up run (the only phase whose FUSE counters cover just the command):
The shared path costs the same everywhere. The daemon's time is within 8% across the four, with the same request counts.
Most of the difference is in C02, C04 and C05, which start about 1,000 short processes each. Their time outside the daemon:
So the gap is process start-up in each VM's kernel, not FUSE.
Docker's seccomp filter is probably not the cause: Lima has none and is as slow as Docker Desktop. This is an inference and isn't isolated.
The C12 finding above is within noise. In the warm-up run of the same body, all three VMs were faster than Docker Desktop on C12 (151–169 ms against 171), the reverse of the timed run.
Conclusion: against the LayerFS Docker Desktop baseline, the no-Docker VMs leave the filesystem path unchanged. Command time then follows how fast the VM's kernel starts processes: Apple and Microsandbox are about 8–9% faster, Lima about 7% slower. Separately, FUSE costs about 9–10× plain container storage on these cells, on every platform.
Script:
native_cell.py(sha256 recorded in each result) next to the runner, with the bodies read fromworkloads.pyat196c714b0b^. No containers or volumes are left.Mola-maker commentedon Nov 1, 2026
感谢提出这个 issue，也感谢 AI 协作工具在整个排查过程中做的分析 🙏
对 AI agent 的使用来说，这个问题的解决会直接改善 agent 选择 sandbox 后端时的量化依据——成本数据直接决定 agent 部署形态。
两点建议：