Repository navigation
fabric/tp: multirank RoCE over a switchless ring, each peer's stripes on the HCAs that reach it - #1516
Conversation
|
Thanks @Bizuayeu. This is the shape we asked for on #1288, and it keeps both rules from #1495's review. We read it through:
Next we'll run |
Done in |
…, so a switchless ring connects roce_multi paired HCA h with every peer's HCA h, which assumes a switch: on a ring of three, where each cable is its own pair of /24s, the connect step refused (BadRecord). A plan now pairs our k-th HCA on a /24 with the peer's k-th on it (roce_multi_pairs.zig), so a switch keeps HCA h to every peer's HCA h over two stripes and a ring gives each peer its link's HCA over one. Every peer gets the same number of stripes, the flags the kernel counts from each source, so unequal stripes, a peer out of reach and an HCA that reaches nobody refuse before any queue pair moves. Named HCAs (TF_TP_ROCE_HCA) are paired the same way. The kernel's n_hca is the stripe count.
…alone With two ranks the pick takes the ports on a /24 the peer shares. With more, one /24 need not reach every peer: on a switchless ring each neighbour sits on its own link. choosePeers runs that pick toward each peer alone and keeps the most ports a peer whose union fits, so a ring picks one port toward each neighbour and a switch the same two ports to every peer.
… peers' gathers at the timeout Three ranks on host threads over the shm wire: the last stops after ten gathers, and the others' next gather must return error.Timeout once the timeout has passed, and well before ten of them. A watchdog aborts the wires when the timeout does not end the wait, so the test fails with error.Aborted instead of hanging; without the timeout check in hostGatherAt it does.
…lure, and a test reads the report The report is unchanged; as a function, a test can read it (logs are quiet under test). The test writes the kernel's record of a wait that timed out, seq 7 from rank 2, and expects error.PeerFailed and a report naming rank 1's wait for seq 7 from rank 2. A report built from other numbers fails it.
…e rank's mismatch drops the wire on every rank The vote is unchanged; as its own function it runs on host threads over a host-only wire. Three ranks vote with rank 1's probe failing, and every rank ends without the wire, falling back to NCCL; with every probe matching, every rank keeps it. A vote on each rank's own probe alone leaves ranks 0 and 2 on the wire and fails the test.
… pins a CPU and maps /dev/shm
6256add to
f3feba5
Compare
|
Thank you @Bizuayeu. This is in TensorFold 1.2.0, released today: https://github.com/ashhart/TensorFold/releases/tag/v1.2.0 It landed with your commits under your own name, so you're listed as a TensorFold contributor, and it's credited to you in the release notes and the changelog. |
What this changes
On a switchless ring of three, where each cable is its own pair of /24s,
roce_multirefused at connect (BadRecord, #1288): it paired HCA h with every peer's HCA h, as on a switch. Each peer's stripes now go over the HCAs that share its /24 (roce_multi_pairs.zig): a switch keeps HCA h to every peer's HCA h over two stripes, as before, and a ring gives each peer its link's HCA over one. With three ranks or more,roce_pickpicks ports toward each peer alone. Every peer gets the same number of stripes, since the kernel counts that many flags from each source, so unequal stripes, a peer out of reach and an HCA that reaches nobody refuse before any queue pair moves. Named HCAs (TF_TP_ROCE_HCA) are paired the same way. The record, the layout, the kernel and the two-rank wire are unchanged; the kernel'sn_hcais now the stripe count, which on a switch is the HCA count as before.It also adds the three tests #1288 asked of the multirank wire, each with a case that fails without its property:
roce_multi_wait_test.zig: three ranks on host threads over the shm wire, with a watchdog that turns a hang intoerror.Aborted);check's report moves, unchanged, intodescribeFailure, which the test reads after writing the kernel's record);keepIfAll, which three host ranks run).The two rules #1495's review set for this move hold here. A rank that refuses still joins the agreement: a
connectthat refuses is a false vote inRoce.init's agreement, as a failed pick is in the session's, not an early return. And nothing is paired by index across peers: every peer's stripes come from the /24s alone, settled for all of them before any queue pair moves, so a peer with no shared /24 refuses (the pick then opens the first usable ports only so that every rank refuses at connect).Part of #1288, route (b), and the
roce_multiside of #1495's review. GLM'stp_groupborrowing this wire follows in its own pull request, opt-in for worlds above two.Receipt
82fa993a6on main13255c97d(the head,6256add49, only keeps two host tests to Linux); three GB10 cabled in a ring of three QSFP cables with no switch, each port seen as two RoCE v2 interfaces on that link's two /24s;tf-tp-testbuilt ReleaseSafe for sm_121.tf-tp-test suiteat world 3 over the ring, roce withTF_TP_ROCE_FALLBACK=error. Main: every rank refuses at connect (BadRecord, thenRoceUnavailable), no check runs. This branch: every rank connects, each peer over its link's HCA (rank 0 to rank 1 on its HCA 0, to rank 2 on its HCA 1), and the suite passes 402, 402 and 328 checks on ranks 0, 1 and 2, every one exact, 0 failed; nccl on the same binary passes 201, 201 and 164, 0 failed. An 8 KiB exchange (decode_window, 86 in a row) takes 10.74 µs a time on the wire against 26.32 µs on NCCL in the same run.test-roce-multi-host40/40,test-tp130/131 (1 skipped: the two-rank verbs run, which needsTF_TP_ROCE_TEST_HCA),compile-config-server7/7,test-cuda-host262/262 steps (3,748 of 3,757 tests, 9 skipped),zig build test428/428 steps (1,580 of 1,596, 16 skipped),zig fmt --check, andtools/zig/lean_check.pywith nothing new (428 against main13255c97d's 428, and 425 against7c0688951's 425 merged with it). Each new test fails with its property taken out, and only that test: without the timeout check inhostGatherAtthe wait test ends in the watchdog'serror.Aborted; a report built fromlast_seqand this rank fails the report test; a vote on each rank's own probe leaves ranks 0 and 2 on the wire and fails the probe test. Every commit passestest-roce-multi-hostandtest-tp. At6256add49the two tests that run the shm wire skip off Linux at comptime: built for aarch64-macos,test-roce-multi-hostcompiles (not run: no Mac here), where82fa993a6failed onsched_setaffinity.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (No family path changes; the suite's ops are byte-equal to NCCL's.)SUMMARYline is in the receipt. (No kernel changes;n_hcacarries the stripe count.)