Document Nix remote builder setup on the tailnet - #198
Draft
samestep wants to merge 3 commits into
Draft
Conversation
Covers the two non-obvious gotchas for reaching a working cross-arch remote-builder state: declaring each builder's system features in /etc/nix/machines (so nixos-test/kvm VM tests get an eligible builder), and disabling Ubuntu's AppArmor unprivileged-userns restriction on the Linux builders (needed by nix/lix functional test suites). Also notes the shared-/tmp wart on the sandbox-less macOS builder.
Each builder runs cores = 0 (one compile job per core), which assumes each compile fits in ~1 GiB. foundationdb doesn't: its Flow actor compiler emits enormous translation units, and with -O3 (and LTO on Linux) a single cc1plus peaks at 3-4 GiB. One job per core then overcommits RAM several-fold, OOM-killing compilers on the Linux builders and thrashing the memory compressor into a crawl on the 16 GiB macOS one. The scarce resource is RAM-per-core, and all three builders are high-core / modest-RAM boxes. Document capping cores to ~RAM/4 GiB per builder (6 / 12 / 3), why no daemon restart is needed (ssh:// serve re-reads nix.conf and doesn't inherit the coordinator's cores), and where a less-coarse fix belongs: a per-package env.NIX_BUILD_CORES in nixpkgs (as mame already does) or, ultimately, memory-aware scheduling in Nix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QzCW6ppPqHqkqNMCkRisLE
Capping cores bounds one job; the real budget is max-jobs × cores, so sizing cores to RAM ÷ 4 GiB while leaving max-jobs alone quietly assumed max-jobs = 1. On the coordinator that was accidentally true — 1 is Nix's default and nothing had set it — so ubuntu built its own aarch64-linux serially at 6 of 18 cores while fanning 8 and 6 jobs out to the remotes, which reads in npb as the local system having far more work left than the others. On sandbox-amd64 it was false in the other direction: max-jobs = 8 with cores = 12 is up to 96 compilers on 32 CPUs. Budget both per machine against its CPU count (jobs × cores ≈ CPUs) and document why overshooting buys contention rather than concurrency, why ubuntu stops two CPUs short as the driver, and why the RAM tail is left unbudgeted: per-job memory isn't normally distributed, so surviving a pair of foundationdb-class builds would mean throttling everything else permanently, where an OOM is a sticky Failed that --retry clears. Also document driving from sandbox-amd64: its machines file pointed aarch64-linux at the deleted sandbox-arm64 VM, and a coordinator's user must be in the target's trusted-users or builds arriving there fail with "not privileged to build input-addressed derivations" — the one setting here that needs a daemon restart. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.