[Bizuayeu]

On three GB10 in a ring (GLM-5.3-Flash XSFP4 at TP=3, one binary served with TF_GLM_ROCE_MULTI=0 and =1, two launches each, the same tokens in every launch):

	NCCL (=0, as main)	the wire (=1)	change
decode with the drafts (tok/s)	51.23, 51.26	52.56, 52.56	+2.57%
decode with --no-drafts (tok/s)	30.51, 30.61	30.76, 31.05	+1.13%
cold 136k-token prompt with the drafts (ttft, s)	83.48, 83.56	84.83, 84.84	1.57% slower
cold 136k-token prompt with --no-drafts (ttft, s)	79.58, 79.97	80.74, 80.80	1.24% slower

What this changes

At three ranks or more GLM's small all-gathers all go to NCCL; the RoCE gather in cuda/roce.zig stops at two. With TF_GLM_ROCE_MULTI=1, each rank borrows fabric/tp's multirank RoCE for the main stream's small all-gathers. tp_group.Group gets a multi wire beside NCCL: allGather sends what fits down it, and check, abort and close reach it. Each rank opens fabric/tp's bootstrap at TF_TP_MASTER (GLM's rank and world in place of TF_TP_RANK and TF_TP_WORLD), agrees the wire's settings, picks its ports, brings up the wire and its kernel (TF_TP_KERNEL_IMAGE, as DeepSeek reads it), and probes one 4 KiB in-place gather on the wire against the group's NCCL. Every step before the probe's gathers ends in a vote over the bootstrap, so a rank that fails at any of them leaves every rank on NCCL (TF_TP_ROCE_FALLBACK=error refuses instead). The probe's buffer, stream and uploads are ready on every rank, by a vote, before any rank enters a gather; then every rank enters both, NCCL's even when its wire gather fails to enqueue. From there a failed NCCL gather or wait refuses whatever TF_TP_ROCE_FALLBACK says, since a peer may be waiting in NCCL, and the ranks' verdict on the bytes is the last vote: only when every rank's probe matched is the wire lent. The settings refuse TF_GLM_ROCE_MULTI=1 without TF_TP_MASTER and TF_TP_KERNEL_IMAGE before the weights load. The wire's waits are fabric/tp's: bounded by time and by its abort flag, and a failed one's check names the rank and the seq it waited on.

It stays opt-in. Decode gains, but a cold long prompt runs 1.2 to 1.6% slower with the wire open, although prefill sends nothing down it. #1288's comments trace that to moving the wire's queue pairs to ready-to-receive on the NIC functions NCCL also uses, a cost that outlives the queue pairs (opening them, or running the proxy, leaves nothing); keeping the two on separate functions removes it but costs NCCL a rail, 9.75% on this ring's cold prompt (two rails, not merged; #1288). So NCCL keeps every world above two by default until a cold-prompt receipt shows no loss.

This builds on #1516, which lets roce_multi connect on a switchless ring. #1516 has landed, so this is now its own six commits on main 581cbe65c. Part of #1288, route (b).

Receipt

• Environment: TensorFold cec391feb (fabric/tp: multirank RoCE over a switchless ring, each peer's stripes on the HCAs that reach it #1516's six commits on main e4cc630f5, then the six here); GLM-5.3-Flash XSFP4 at TP=3, PARALLEL=4, --prompt-cache-gib 0; three GB10 cabled in a ring of three QSFP cables with no switch, each port seen as two RoCE v2 interfaces on its link's two /24s; zig build native for sm_121, and fabric/tp's mailbox.cu built as the kernel image. One binary served with TF_GLM_ROCE_MULTI=1 (and TF_TP_ROCE_FALLBACK=error, so a wire that does not open refuses) and with =0, launched wire, NCCL, NCCL, wire with the drafts and again with --no-drafts.
• Exactness: every wire launch's tokens equal its NCCL pair's (rank 0's done lines without ids and times, 28 in each), and the bench's token hashes agree across all eight launches (both decode prompts and the 136k prompt's three phases). In each wire launch every rank's probe matched NCCL's bytes before the wire was lent, each peer over the HCA on its link.
• Decode speed before and after: the table above; decode is the mean of bench_openai's medians over its two prompts (256 tokens, 3 repeats, temperature 0), and every wire launch beat every NCCL launch. One stream (bench_concurrent): code +3.40% and chat +1.68% with the drafts, +1.64% and +2.83% without; four streams: code +2.54% and chat +0.39% with the drafts, +0.83% and +1.84% without (the launches overlap in chat with the drafts and in code without). After the cold 136k prompt, decode is 2.41 to 3.40% faster with the drafts and 0.48 to 1.51% without.
• Prompt speed before and after: the cold 136k-token prompt (bench_cache --require-cold) is 1.57% slower in its first phase with the drafts and 1.24% without, and 1.3 to 1.5% slower in its repeated and concurrent phases. No launch waited on the engine's heat guard (no heat_wait_s on any rank).
• Tests run (Linux x86_64): test-roce-multi-host 40/40, test-tp 130/131 (1 skipped: the two-rank verbs run, which needs TF_TP_ROCE_TEST_HCA), compile-config-server 7/7, test-cuda-host 268/268 steps (3,825 of 3,834 tests, 9 skipped), zig build test 434/434 steps (1,608 of 1,624, 16 skipped), compile-cuda-tests 10/10 steps (the CUDA runner, which CI does not build), zig fmt --check, and tools/zig/lean_check.py with nothing new (426 against main e4cc630f5's 426, and 425 against 7c0688951's 425 merged with it). The probe test fails with the readiness vote taken out: a failed prepare on one rank leaves another lending the wire.
• Not run, and why: Metal (nothing here reaches it); a switched multirank network (no switch here); a ring of four (a fourth node arrives in early November).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (The wire carries only the group's all-gathers; drafts on and off are both in the receipt.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (The token hashes agree across the drafted and plain launches, on the wire and on NCCL. The prompt cache was off and the four-stream runs were not hashed; the wire only copies bytes, so resumed and concurrent prompts take the same sums as on NCCL.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No bits change: the wire's gathers are byte copies, and every launch's tokens match.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (No kernel changes; the wire runs fabric/tp's mailbox kernel as it is.)
• Prompt processing is no slower than the last release. (Off by default, the path is main's; with it on, the cold prompt's cost is in the table.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (CUDA on three GB10 and the host tests on Linux; Metal not run, as above.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (The probe test fails without the readiness vote, as above; it runs on host threads and skips off Linux.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[Bizuayeu]

Rebased on main c9bfbfad4, which this branch conflicted with from the start: main's 91dfa11e8 moved glmCuda into zig/build/cuda_engines.zig, so the build commit's one line (tp over the engine's own cuda module) now sits there. The other ten commits are unchanged (git range-diff), and the receipts in the description are from 2ceff2601, before the rebase. On the new head a634d64d0: test-roce-multi-host 40/40, test-tp 130/131 (one skipped), compile-config-server 7/7, compile-cuda-tests 10/10, zig fmt --check, lean 0 new against main; test-cuda-host and zig build test fail only Qwen3.5's cuda_cost_identity, which fails the same way on main here.

[ashhart]

Thanks @Bizuayeu. GLM still uses NCCL above two ranks by default, and no GPU source changes, so this is host code only. Merged with main, the Linux server and runner compile. Two things before we run it on device:

1. Vote on probe preparation before the collectives. In zig/src/families/glm/cuda_roce_multi.zig:48, probe(...) catch false reaches lendIfAll only after the probe returns. If one rank fails DeviceBuffer.alloc, Stream.init or the upload (:58, :60, :64), it skips its NCCL allGather at :67 while its peers have already enqueued theirs. A timeout and local fallback can't bring the NCCL sequences back in line after that. Please prepare everything, vote readiness on every rank, and only then enter both collectives. A failure after enqueue that can't fall back safely should fail the group. A test for each failure point on each rank, requiring the same refusal or fallback everywhere, would pin it; today's probe-vote test only injects finished probe results.
2. Mac host tests. test-tp and test-roce-multi-host no longer compile on a Mac. They reach proxyLoop → pin, which uses std.os.linux.sched_setaffinity (roce_multi.zig:548), and they need /dev/shm. These come with fabric/tp: multirank RoCE over a switchless ring, each peer's stripes on the HCAs that reach it #1516's commits, so please keep those tests to Linux there.

This branch carries #1516's first five commits, so we'll land #1516 once and then this on top.

[Bizuayeu]

Please prepare everything, vote readiness on every rank, and only then enter both collectives. A failure after enqueue that can't fall back safely should fail the group. A test for each failure point on each rank, requiring the same refusal or fallback everywhere, would pin it; today's probe-vote test only injects finished probe results.

#1524 (comment)

Done in cec391feb, and the branch is rebased on main e4cc630f5. #1404 carried the same commit that moves the ranks' checks to their own file, so that commit drops out here, and cuda_ranks_test.zig keeps main's TF_GLM_INDEX_SPLIT test beside this branch's TF_GLM_ROCE_MULTI one. The branch is now #1516's six commits (its five, then the Linux gate for its shm-wire tests, answered on #1516) followed by the six here.

• The probe runs in steps. prepare makes the buffer and stream and finishes both uploads before any gather, and every rank votes it ready over the bootstrap. If any rank is not ready, no rank enters a gather and every rank keeps NCCL.

• Then every rank enters both gathers. A rank whose wire gather fails to enqueue still enqueues NCCL's, so NCCL's sequence stays the same on every rank, and that rank votes no.

• A rank whose NCCL gather or wait fails returns at once without voting, since its peers may be waiting on NCCL. Once a rank has enqueued NCCL's gather, open falls back only on every rank's verdict (ProbeMismatch), and refuses on anything else whatever TF_TP_ROCE_FALLBACK says. The refusal fails that rank's startup, and the ranks' watch stops the others.

• The old test injected only finished results; the new one replaces it. It runs three ranks on host threads with each step failing on each rank in turn: 15 runs, plus one with no failure. Every rank must reach the same outcome:

  • keep NCCL when prepare, the wire's gather or the verdict fails;
  • refuse when NCCL's gather or the wait fails;
  • lend the wire when nothing fails.

  When prepare fails, no rank enters either gather; otherwise every rank enters NCCL's. With the readiness vote taken out, the test fails.

• On our ring of three GB10 at TP=3 the body's receipt is new, from cec391feb: the eight launches again (the wire against NCCL, the drafts on and off). In every wire launch each rank's probe matched NCCL's before the wire was lent, and every launch's tokens are the same. Decode gains as before (+2.57% with the drafts, +1.13% without), and the cold 136k prompt is 1.2 to 1.6% slower.

• Tests (Linux x86_64): test-roce-multi-host 40/40, test-tp 130/131 (1 skipped), compile-config-server 7/7, test-cuda-host 268/268 steps, zig build test 434/434 steps, compile-cuda-tests 10/10, zig fmt --check, and lean_check.py with nothing new (426 against main e4cc630f5's 426, and 425 against 7c0688951's 425 merged with it).

[ashhart]

Hi @Bizuayeu, a status update on #1524.

#1516 landed tonight, so this branch can now sit on main. We read its own diff on top of #1516. The default path
is unchanged: the setting defaults off, and with it off the only added work is a null check.

Could you rebase it onto current main? After that, the run still owed needs two GB10s at once: GLM at TP=2 with
the setting unset, against main (token-equal and no slower). Your TP=3 receipts cover the opt-in path, so we only
need to repeat it if three RoCE-linked GB10s are free at the time.

[Bizuayeu]

Could you rebase it onto current main?

(#1524 (comment))

Rebased onto main 581cbe65c: the six commits after #1516, with no conflicts. On the files this PR touches, the tree differs from the old head only by main's own draft_kind lines in cuda_native.zig (3dfd41f).

Checks on the rebased head (Linux x86_64): test-roce-multi-host 40/40, test-tp 130/131 (1 skipped), test-cuda-host 3,910 of 3,920 and zig build test 1,674 of 1,692 (the one failure in each is cuda_cost_identity's timing-dependent drafter test, outside this PR's files), compile-config-server 7/7, compile-cuda-tests 10/10, zig fmt --check, and tools/zig/lean_check.py with nothing new (425 against main's 425).

If it helps, I can take the TP=2 run here on two GB10s (the setting unset, against main: token hashes and decode).

[ashhart]

Thank you @Bizuayeu. This is in TensorFold 1.2.0, released today: https://github.com/ashhart/TensorFold/releases/tag/v1.2.0

It landed with your commits under your own name, so you're listed as a TensorFold contributor, and it's credited to you in the release notes and the changelog.