Conversation
cijohnson
force-pushed
the
ichristo/cvs-spur-rccl-tests
branch
6 times, most recently
from
September 11, 2026 04:45
9cc0c5f to
774093d
Compare
of mpirun on managed clusters. One HTTP agent per node plus local UDS workers spawn children that inherit the scheduler PMIx slot, so perf/regression can run under srun --mpi=pmix without a hostfile. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Ignatious Johnson <ichristo@amd.com>
starts its child(MPI program).
After create_subprocess_exec, the coordinator records its own spawn(MPI program)
and each UDS worker writes {"kind":"spawned","rank":N} on the existing launch.sock.
The node waits up to 30s for every local rank; if any of it is missing it cancels
children(MPI program) and returns HTTP 500 so rank 0's existing launch fan-out
cancels the other nodes.
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Ignatious Johnson <ichristo@amd.com>
A dead extra rank used to vanish with except/pass, then launch waited 60s for _ready. Protocol errors now go through globals.log, and once the node has seen a full worker set a short map fails with local UDS workers have/expected instead of blocking. Co-authored-by: Cursor <cursoragent@cursor.com>
cijohnson
force-pushed
the
ichristo/cvs-spur-rccl-tests
branch
from
September 11, 2026 04:59
2368435 to
8c8e338
Compare
A {"kind":"cancel"} between launches was parsed as LaunchRequest and killed the worker. Skip it while idle; in-flight cancel is still handled during a launch.
Signed-off-by: Ignatious Johnson <ichristo@amd.com>
A boolean launch_busy could let two POSTs both start children on the same node. Hold a lock for the launch and return 409 if it is already held. Signed-off-by: Ignatious Johnson <ichristo@amd.com>
Skip mpirun and launch the same argv in the srun PMIx slots. Also remove unused Rank0Runner._task_world_size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Technical Details
Test Plan
Test Result
Submission Checklist