[luthermonson]

Why

• Host-wide OOMs. sharks (40 GB, 5 slots) had global OOMs on 6 of the last 14 days. On 2026-10-23 at 03:00Z one OOM killed processes in three concurrent ephpm jobs at once, including a 2 GB ld. A per-job memory.max confines an OOM to the job that caused it.
• Leaked cgroups. Every finished job leaves an empty /sys/fs/cgroup/ephemerd-dind-<id> behind: 477 on ducks and 541 on sharks today.

Memory budget (opt-in, off by default)

[runtime]
memory_limit_mb = 16384      # 0/unset = no limit (unchanged behaviour)
memory_high_percent = 90     # memory.high = 90% of the limit; 50-100, 100 = no soft limit

It's a per-JOB budget, not per container. The heavy work in a container: job runs in a dind sibling, so a per-container limit would let a job use N × the limit.

• How it works: siblings already share one parent cgroup, /ephemerd-dind-<id>, which containerd derives from the job's dind namespace. With a budget set, the runner container moves under that same parent (/ephemerd-dind-<id>/runner). The parent carries memory.max plus memory.high, so page cache is reclaimed and the job throttled before anything is killed.
• The alternative is ruled out: siblings can't nest under the runner's own cgroup, because cgroup v2's no-internal-processes rule forbids a cgroup that has processes from having controller-enabled children.
• What an OOM looks like: the kernel kills inside the job, usually the largest process (a linker), so the step fails. memory.oom.group is deliberately NOT set: killing the runner too would turn a failed step into a "runner lost" job.
• Fail-open: if the parent can't be created or limited, the job runs unlimited, as before, with a WARN.
• Self-healing limits: the parent's limits are rewritten before every container of the job, so they hold even if the sweep below removed an empty parent mid-job.
• Limitation: docker build RUN steps run in BuildKit's own cgroup and aren't covered.
• Linux only; Windows and macOS have no-op stubs.

Leaked cgroups

runc creates the job parent with MkdirAll and only ever removes container leaves.

• Teardown: destroySteps now removes the parent after the task delete and Dind.Stop.
• Sweep: the node disk sweeper (at startup and on every tick) removes ephemerd-dind-* cgroups that have no processes and no child cgroups and have been idle for 10 min. It never touches anything else.

New package pkg/jobcgroup owns the layout. A dind test pins jobcgroup.Prefix == DindNamespacePrefix.

Verification

• go build ./....
• go vet passes on Windows and with GOOS=linux.
• go test passes on Windows for config, dind and runtime.
• The Linux test binaries pass under WSL Ubuntu:
  • pkg/jobcgroup: budget written and resized, no budget creates nothing, the memory controller enabled, Remove drops empty but keeps busy, and the sweep keeps cgroups with procs or children, others' cgroups, and ones inside the grace period;
  • runtime Rlimit|Pids|JobCgroup, including fail-open;
  • the full pkg/dind suite, plus TestJobMemoryOpts.
• New config tests cover resolution and validation (it rejects a value below 512 MB, e.g. GB typed as MB).

Not verified on metal yet: writing memory.max on a live node's cgroupfs, and runc accepting the pre-created parent. The suggested rollout is one node first, e.g. sharks with memory_limit_mb = 12288 and 4 slots, after a few days of per-job anon-memory sampling.