feat(aorta): collect per-node traces, survive partial node failure - #331
Merged
speriaswamy-amd merged 6 commits intoSep 11, 2026
Merged
Conversation
5 of 6 tasks
speriaswamy-amd
force-pushed
the
surya/aorta-mn-06-traces
branch
from
September 8, 2026 15:18
b20751e to
8897ea8
Compare
speriaswamy-amd
force-pushed
the
surya/aorta-mn-06-traces
branch
from
September 8, 2026 15:30
8897ea8 to
0261169
Compare
amd-droy
reviewed
Sep 9, 2026
Two problems with profiler artifacts on a multi-node run. First, each node wrote its torch_profiler/ tree to its own filesystem, so the host parser only ever saw the head node's ranks. _collect_multi_node_traces() consolidates every node's trees into <aorta_path>/combined_traces/node_<rank>/: a local copy when the orchestrator shares the head's filesystem, rsync over SSH otherwise, scp -r where rsync is absent. Per-node failures are logged and skipped rather than aborting the collection. The trace-discovery scan skips anything under combined_traces/ so the stale per-node originals cannot shadow the consolidated set, and seeds trace_mtime from the collected tree - without that seed the existing freshest-trace comparison raises UnboundLocalError once trace_dir can be pre-set. Second, run() returned as soon as any node failed, before collecting anything. One flaky node in a multi-hour run therefore discarded every surviving node's traces, forcing a full rerun or a manual TraceLensParser salvage. Collection and artifact discovery now run unconditionally; the run is still reported FAILED or TIMEOUT with the offending nodes named in error_message, but artifacts come from whatever the survivors produced. Co-Authored-By: Claude <noreply@anthropic.com>
…ace dirs combined_traces_in() must be checked against aorta_path directly, not an intermediate combined_root variable that pointed at the wrong directory.
…ult.succeeded TraceLensParser and AortaReportParser both refused to parse whenever a run was not COMPLETED, undermining trace collection surviving partial node failure. Now they only bail out if there is genuinely nothing on disk to parse, and record the run failure as a warning otherwise.
…n partial failure - Clear a node's combined_traces dest before repopulating it, so a node that fails to produce fresh traces this run can't have stale prior-run data mistaken for current-run data. - Resolve the TraceLens/GEMM analysis output_dir to the head node's own torch_profiler tree when trace_dir is the aggregated combined_traces root, instead of passing the whole aorta_path mount. - Preserve already-collected artifacts on RunResult when run() hits an exception after trace collection, instead of dropping them. - Isolate the TraceLens dependency probe in its own try/except so a failure probing for it can't be mistaken for a failure of the run. - Report the run's actual status in the generated benchmark report instead of hardcoding "completed". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d cluster resize - _collect_multi_node_traces now wipes and recreates combined_traces from scratch each run, so a shrunk cluster no longer leaves a previous larger run's higher-numbered node_<rank> directories for the parser to misread. - _copy_local_torch_profilers/_copy_remote_torch_profilers accept a min_mtime floor and skip torch_profiler trees with no file modified at or after it, so a node that fails before writing new traces no longer has its stale previous-run output copied in as if it were current. run() derives the floor from its own start_time with a clock-skew tolerance. - test_parse_results only prefers container-generated Excel reports on single-node runs, since TraceLens analysis only ever covers the head node's container; on multi-node runs the complete raw-trace parse stays authoritative so metrics aren't silently scoped to the head node. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…from reintroducing stale traces - _copy_local_torch_profilers/_copy_remote_torch_profilers now filter individual files by mtime instead of skipping or keeping a whole torch_profiler tree, so a reused output dir with a mix of old and new rank files only copies the new ones. - _copy_remote_torch_profilers lists remote files with mtimes via find, filters by freshness, then transfers the selected files in one bulk rsync (falling back to per-file scp), instead of rsyncing/scp-ing whole directories. - run() no longer falls back to scanning aorta_path for any torch_profiler directory (including the hardcoded legacy path name) once multi-node trace collection has already run; that fallback ran unconditionally before and could reintroduce a stale trace that collection had just filtered out. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
speriaswamy-amd
force-pushed
the
surya/aorta-mn-06-traces
branch
from
September 10, 2026 23:13
0261169 to
8cd5bde
Compare
speriaswamy-amd
removed this pull request from stack #333
September 11, 2026 00:23
speriaswamy-amd
added this pull request to stack #414
September 11, 2026 00:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 6/6 — splits #171. Base: #330. Final PR; the tip of this stack is identical to #171's tree.
Why
Two problems with profiler artifacts on a multi-node run.
torch_profiler/tree to its own filesystem, so the host parser only ever saw the head node's ranks.run()returned as soon as any node failed, before collecting anything. One flaky node in a multi-hour run discarded every surviving node's traces, forcing a full rerun or a manualTraceLensParsersalvage.What changed
_collect_multi_node_traces()— consolidates every node's trees into<aorta_path>/combined_traces/node_<rank>/: local copy when the orchestrator shares the head's filesystem, rsync over SSH otherwise,scp -rwhere rsync is absent. Per-node failures are logged and skipped rather than aborting the collection.combined_traces/so the stale per-node originals cannot shadow the consolidated set, and seedstrace_mtimefrom the collected tree — without that seed the existing freshest-trace comparison raisesUnboundLocalErroroncetrace_dircan be pre-set.FAILED/TIMEOUTwith the offending nodes named inerror_message, but artifacts come from whatever the survivors produced.Test
ruffclean. Unit tests 631 → 638 (7 new:combined_tracespath predicate, local copy-tree with recursion guard, collection layout, and partial-failure-still-collects).Live validation from #171 still applies unchanged — this stack's tip is byte-identical to that branch for every non-test file:
cvs run test_aortaon a real 2-node cluster (g17u19 + f16u13, 16×MI300X), 5/5 pytest cases in 148s, traces from both nodes, host parser produced metrics for all 16 ranks.