feat(aorta): disaggregated multi-node torchrun launch (AIMVT-173) - #329
Closed
speriaswamy-amd wants to merge 0 commit into
Closed
speriaswamy-amd wants to merge 0 commit into
speriaswamy-amd wants to merge 0 commit into
Conversation
This was referenced Aug 14, 2026
speriaswamy-amd
force-pushed
the
surya/aorta-mn-03-config
branch
from
September 8, 2026 15:18
560d01e to
ae91e9c
Compare
speriaswamy-amd
force-pushed
the
surya/aorta-mn-04-torchrun
branch
from
September 8, 2026 15:18
211be2b to
7ba6f81
Compare
speriaswamy-amd
force-pushed
the
surya/aorta-mn-03-config
branch
from
September 8, 2026 15:28
5cde8b9 to
8460ab0
Compare
speriaswamy-amd
force-pushed
the
surya/aorta-mn-04-torchrun
branch
from
September 8, 2026 15:28
7ba6f81 to
8460ab0
Compare
Collaborator
Author
|
Reopening — this was auto-closed by GitHub when a rebase mistake on my end briefly collapsed this branch to be identical to its base (0 commits/0 diff). The branch has since been corrected and force-pushed with the real commit restored; verified diff is +564/-42 across 2 files as intended. |
Collaborator
Author
|
Superseded by #390 (same branches, same content) — this PR got stuck in an auto-closed state that GitHub wouldn't let me reopen after a rebase mistake temporarily zeroed its diff. No content was lost. |
speriaswamy-amd
removed this pull request from stack #333
September 11, 2026 00:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 4/6 — splits #171. Base: #328. Implements AIMVT-173.
Why
Run the Aorta benchmark across N nodes from a single
cluster.jsoninstead of requiring N single-node cluster files, mirroring the disaggregated pattern the PyTorch xDiT and SGLang multi-node suites already use.What changed
run()branches on the resolved launch mode:script— today's behavior verbatim (experiment_scripton the head node).torchrun— a rank-group on every node in parallel, rendezvous-ing on the head.auto(default) —scriptfor 1-node clusters,torchrunabove that. Existing single-node configs are unaffected.New helpers in
cvs/runners/aorta.py:_resolve_launch_mode()—scriptwith >1 node is a config error rather than a silent single-node run._resolve_master_addr()— explicit override → head node's VPC/RDMA address → plain node id. The mgmt/SSH address is frequently orchestrator-only and unreachable from peer nodes._pick_free_port_on()/_pick_master_port()— binds and releases a port over SSH on the head node, which is the host that actually holds the rendezvous socket. The snippet is passed as oneshlex-quoted argument; as separate argv elements the remote shell re-splits it on its own semicolons._build_torchrun_command(),_run_single_node().TraceLens/GEMM analysis now explicitly target the head node's container, and training-log discovery iterates the node list, rather than relying on locals left over from the single-node path.
Test
ruffclean. Unit tests 603 → 623 (20 new: mode resolution, port/addr selection, command construction,validate_config).