Repository navigation
fix(runtime): per-job cgroup pids.max instead of a host-wide shared RLIMIT_NPROC - #218
Merged
Merged
Conversation
…T_NPROC
RLIMIT_NPROC is counted per UID across the whole host, and every runner
container runs as the same UID with no user namespace, so concurrent jobs
on a node drew from one shared 1024-task budget. On ducks (2026-10-22,
4 concurrent jobs) two ephpm builds failed with rustc panics:
failed to spawn work thread: Os { code: 11, kind: WouldBlock,
message: "Resource temporarily unavailable" }
(runs 39785206572 and 39863141372; uid 1001 held 558 threads at an idle
moment). No OOM kills, no kernel messages, and kernel.threads-max was not
close -- a hit RLIMIT_NPROC fails clone() silently.
- [runtime] pids_limit sets the cgroup pids.max on the runner container
and on every dind sibling of its job. Default 32768 per container; -1
disables. A cgroup limit counts per container, so a fork bomb in one job
cannot starve another and a busy neighbour cannot fail a build.
- The default RLIMIT_NPROC rises from 1024 to 1<<20: as a per-job limit it
never worked, it only made jobs fail each other. An explicit
[runtime.rlimits] nproc is still honoured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ePHPm hit two rustc internal compiler errors on 2026-10-22, both on ducks with 4 concurrent jobs:
write.rs:1276,failed to spawn helper thread: Os { code: 11, ... "Resource temporarily unavailable" }write.rs:1915/2157,failed to spawn work thread: ... code: 11EAGAIN from thread creation means a task-count limit, not memory. ducks has no
[runtime.rlimits], so every runner container getsRLIMIT_NPROC=1024. The kernel counts that per UID across the host, and every runner container runs as uid 1001 with no user namespace. So all concurrent jobs share one 1,024-task budget. uid 1001 held 558 threads at an idle moment with 4 jobs, and parallel rustc/LLVM codegen spikes push it over. There were no OOM kills and no kernel messages (a hit RLIMIT_NPROC failsclone()silently).kernel.threads-max(385k) was nowhere near.Fix
[runtime] pids_limitsets the cgrouppids.maxon the runner container and every dind sibling of its job. The default is 32,768 per container;-1disables it. It's counted per container, so one job's fork bomb can't starve another, and a busy neighbour can't fail a build.RLIMIT_NPROCgoes from 1024 to 1<<20. As a per-job bound it never worked; it only let jobs fail each other. An explicit[runtime.rlimits] nprocis still honoured.Memory: Linux job containers have no per-container memory limit today. That's unchanged here, and it's worth a separate decision.
Verification
go vetpasses on Windows andGOOS=linux.go test ./pkg/configpasses.Rlimit|Pids(including the newTestPidsLimitOpts) and the fullpkg/dindsuite.TestRuntimeConfig_ResolvedPidsLimit; the NPROC default tests are updated.