[luthermonson]

Problem

ePHPm's nightly "Build (PHP x, macos-aarch64)" fills the macOS job VM to 98–99% during the PHP-linked debug test build. Each test binary embeds libphp + ICU. It has failed with "No space left on device" every night since at least 2026-10-14. ePHPm#634 deletes target/debug after the test step, but the test step itself still runs out mid-link. Every job boots from an APFS clone of base.img (50,000,000,000 bytes), so the guest gets about 41 GiB.

Change

[vm.macos] disk_size_gb (GiB). The default 0 keeps today's behavior.

1. Host, before boot: os.Truncate extends the job's clone. It's sparse on APFS (no host disk is used until the guest writes), and the blocks shared with base.img stay shared.
2. Guest, in the SSH setup step, before the runner starts: diskutil repairDisk <whole> moves the GPT backup header to the new end of the disk, then diskutil apfs resizeContainer <store> 0.
   • The physical store is read from diskutil info /System/Volumes/Data, with / as a fallback. It isn't hard-coded, and it's validated against ^disk[0-9]+s[0-9]+$ before going into the command.
   • The step runs once per VM, even if the SSH setup is retried.
   • A status line with the before and after sizes is logged.

Verified layout of the fleet's base image (hdiutil attach -nomount -readonly + diskutil list, 2026-10-22): disk5s1 Apple_APFS_ISC 524.3 MB, disk5s2 Apple_APFS 44.1 GB, then 5.4 GB of free space. There's no Recovery partition after the container, so it can grow in place.

Failure behavior: warn and carry on.

• A size not larger than the base image is ignored with a WARN at startup. The disk is never shrunk.
• A failed truncate, layout read or resize is a WARN, and the job runs on the base-size disk.
• Each guest command has a 90 s deadline, inside the 5 m provisioning timeout.

Homebrew: the same one-shot step runs brew untap --force buildkite/buildkite as admin (the brew owner on Tart images), if the tap is present. Nothing uses it, and every brew call in ePHPm's nightly warns that it's untrusted. It's best-effort, logged at debug level, with a 60 s cap.

Leaked clones: the Mac's vm/macos/jobs held a 38 GB .img from 2026-10-01 that was never removed.

• Root cause: a VM deleted its clones in exactly one place, at the end of Stop, after the Virtualization.framework stop returned. That stop can hang forever (macOS: the provisioning-deadline self-heal reclaims the VM but LEAKS the concurrency slot — node goes silently dead #196). When it does, the scheduler abandons the VM and the deletion never runs. A daemon restart or crash mid-job leaks the same files: the guest dies with the process and nothing comes back for them. Which of the two hit the Oct 1 job can't be told from code alone, so every path is closed:
  1. Stop runs the guest stop on its own goroutine and removes the files when it returns or after 45 s, whichever is first (teardownMacVMFiles). Unlinking under a wedged guest is safe: it keeps its descriptor. Stop still blocks until the guest is gone, so the scheduler's abandoned-VM accounting (which decides whether another VM may start) is unchanged.
  2. Start removes the clones itself on any error before the VM exists.
  3. A live-VM registry in pkg/vm registers a VM in NewMacOSVM and unregisters it when Stop completes. It covers VMs that are provisioning or stuck stopping, which s.running doesn't.
  4. A periodic sweep in the scheduler runs at startup (kicked by SetMacOSVMConfig) and every 30 min. It removes only jobs/ entries that no live VM owns and that haven't been written for 2 h. It never touches anything outside jobs/ (base.img, aux.bin). It logs each removal with its allocated size (not the sparse apparent size). Totals appear in status as macos_clones_swept / macos_clone_bytes_swept.
  5. ephemerd doctor used to remove only the share directories. It now uses the same sweep with no live set or grace (it only cleans with the daemon stopped), so the disk clones go too.

injectRunnerIntoClone (which mounts the clone) has no callers, so the truncate can't interfere with a host-side mount.

Verified vs. not

• Verified: the base image's partition layout, on the Mac.
• Unit tests:
  • the truncate decision (grow, never shrink);
  • ResolveMacOSDiskSize against a sparse 50e9-byte file;
  • diskutil info parsing, from a fixture built from the verified layout, plus refusal of whole disks and injection attempts;
  • the grow script, and that the guest steps fit inside the provisioning deadline;
  • that the brew script never runs as root;
  • a wedged stop (one that never returns) still removing the clones and share at the bound;
  • a clean stop;
  • the live-VM registry;
  • the sweep's live-VM exclusion, its 2 h grace and its doctor mode, and that it leaves base.img and aux.bin alone;
  • the scheduler sweeper: idle without macOS, passing grace and the live set, the totals, and the startup kick;
  • the config field parse.
• Run on:
  • go test ./pkg/vm ./pkg/scheduler ./pkg/config on Windows, plus linux test binaries under WSL;
  • golangci-lint run (v2.11.4), clean for both GOOS=linux and windows;
  • go vet on Windows and GOOS=linux.
• Darwin: vet needs cgo (vz), so darwin compilation relies on CI.
• NOT verified on metal:
  • the guest-side diskutil repairDisk / resizeContainer commands and the brew untap. They need a canary job on the Mac, and neither can fail a job;
  • the 45 s file removal under a real wedged Vz stop. It's covered only by the fake in TestTeardownMacVMFiles_WedgedStopStillRemovesFiles.

Rollout

mayfly already renders [vm.macos] disk_size_gb (VMSpec.DiskSizeGB → writeVMSection), so only config is needed: [pools.mac-arm64.vm.macos] disk_size_gb = 100, then a release and mayfly node reconfigure on the Mac. The Mac has 232 GiB free and runs one macOS VM at a time.