[Bizuayeu]

On GB10 (GLM-5.3-Flash XSFP4, one binary served with TF_GLM_L2PF=0 and with =1 at the default knobs, which a
GB10 now runs unset, two launches each, the same tokens in every launch):

	off (=0, as main)	on (=1)	change
TP=3, decode with the drafts (tok/s)	51.22, 51.23	52.77, 52.78	+3.02%
TP=3, decode with --no-drafts (tok/s)	30.55, 30.57	31.78, 31.82	+4.06%
TP=3 on #1524's wire, with the drafts (tok/s)	52.31, 52.60	54.49, 54.55	+3.93%
TP=3 on #1524's wire, --no-drafts (tok/s)	30.98, 31.01	32.01, 32.12	+3.44%
TP=2, decode with the drafts (tok/s)	40.38, 40.39	40.39, 41.00	+0.76%
TP=2, decode with --no-drafts (tok/s)	24.40, 24.48	24.89, 24.89	+1.85%

What this changes

GLM's decode streams each layer's weights from memory while its gathers and the KDA chain run. With TF_GLM_L2PF, a
side stream reads the next weights into L2 in those gaps, at three sites a layer on a decode window: before gather_b
the next layer's mixer input (KDA's q, k and v scales, codes and bf16 rest, or DSA's proj and q_b), before gather_a
this layer's ffn head (the router, or a dense layer's gate and up), and between KDA's projections and its chain its o.
A site's rows are the weights' own allocations after they tile, taken in order within TF_GLM_L2PF_MB (12 MiB) as
DeepSeek V4.1's take does, so no row reads past an allocation; a source that starts no allocation turns the prefetch
off. The side stream writes nothing, joins the main stream once a window, and never runs in a prompt pass.

The kernels are DeepSeek V4.1's: l2pf's segments on TF_GLM_L2PF_CTAS (2), and l2pace for the two sites beside a
gather at TF_GLM_L2PF_PACE_GBPS (150; 0 runs segments there too), built into GLM's own images from DeepSeek's
sources. Their two launches get nvcc 13.0 records (glm_l2pf_pending.zig), and the contract runner learns the two
image names. Every rank agrees on the setting and its knobs (the ranks' fingerprint).

It is on by default on a GB10 (sm_121), where the receipts were taken, and off elsewhere until someone measures it:
TF_GLM_L2PF=1 turns it on on any GPU and 0 off. DeepSeek V4.1's prefetch stays opt-in on main. Here every TP=3 row
gains 3 to 4%; at TP=2 decode gains 1.85% without the drafts and 0.76% with them, under a 1% bar for that row, though
no launch was slower. Unpaced (TF_GLM_L2PF_PACE_GBPS=0) gives +1.08% and +1.20% at TP=2 and +2.07% and +1.44% on
the wire at TP=3, so the paced default is the better one at three ranks and close at two.

Three commits: the prefetch (opt-in), its contract records, and the GB10 default.

Receipt

• Environment: TensorFold 6d612760c (glm cuda: at three ranks or more, TF_GLM_ROCE_MULTI=1 sends the small all-gathers over fabric/tp's multirank RoCE #1524's cec391feb on main e4cc630f5, then the three here). The launches ran
  6af36809e, the first of the three, with TF_GLM_L2PF=1, which is what the GB10 default runs; the next two add
  the records and the default. 2ff12c521 is the same three on main ab4b5d8e6, where glm cuda: at three ranks or more, TF_GLM_ROCE_MULTI=1 sends the small all-gathers over fabric/tp's multirank RoCE #1524 and glm cuda: TF_GLM_KV_SHARE keeps prompts as the pool's pages in prompt_radix's store #1732 have landed, but for
  lines of context in contract_limits.zig, cuda.zig, cuda_engine.zig and registry.zig and the landed_cuda line, which takes
  main's exl3_grouped_records term, so the GPU receipts were not taken again. GLM-5.3-Flash XSFP4, PARALLEL=4, --prompt-cache-gib 0, zig build native for sm_121.
  TP=3 on three GB10 cabled in a ring of three QSFP cables with no switch, on NCCL (TF_GLM_ROCE_MULTI unset) and on
  glm cuda: at three ranks or more, TF_GLM_ROCE_MULTI=1 sends the small all-gathers over fabric/tp's multirank RoCE #1524's wire (=1); TP=2 on two GB10 joined by one QSFP cable. Launches alternate: at TP=3 on NCCL off, on, on, off
  with the drafts and again with --no-drafts; on the wire and at TP=2, four arms (off, segments on 2 and on 4 CTAs,
  the default) as 0 1 2 3 3 2 1 0, with the drafts and again without. Each rank logs TF_GLM_L2PF: 226 rows, 1026 MiB a decode window (1,026 to 1,066 MiB by rank and TP).
• Exactness: every launch's tokens equal its off pair's (rank 0's done lines without ids and times, 25 in each), and
  the bench's token hashes agree across all launches of each set (8 at TP=3 on NCCL, 16 on the wire, 16 at TP=2).
• Decode speed before and after: the table above; decode is the mean of bench_openai's medians over its two prompts
  (256 tokens, 3 repeats, temperature 0). One and four streams (bench_concurrent): +1.7 to 4.7% at TP=3 on NCCL,
  +2.6 to 4.6% on the wire, +0.4 to 2.1% at TP=2. At TP=3 every on launch beat every off launch in decode; at TP=2 one
  launch with the drafts tied (40.39 against 40.38 and 40.39).
• Prompt speed before and after: n/a, the prefetch runs on decode windows alone (Forward.prefetch is null in a prompt
  pass), so a prompt runs main's path. No launch waited on the engine's heat guard (no heat_wait_s on any rank).
• Kernel contract (CUDA): kernel-contract-limits on nvcc 13.0.88's fatbins for sm_80, 86, 89, 120 and 121: both
  launches PASS on every SM with their records current (l2pf's segments_kernel<true> 30, 28, 28, 28 and 28
  registers, bound 256; l2pace's paced_kernel 24, 23, 23, 20 and 20, bound 32; sm_121's match DeepSeek's records of
  the same kernels). On the GB10, zig build kernel-contract (with kernel contract: the CUDA runner maps fp8_experts' image, so a run reaches its summary #1579's mapping, without which the run stops at
  fp8_experts) passes both live: l2pf 128 threads of the 256 the driver allows, 28 registers, and l2pace 32 of 32, 20
  registers, as recorded. Its summary: SUMMARY 433 kernels: 433 pass gates 1-4, 0 fail, 6 known findings; 316 controls: 316 red as planted, 0 missed; 183 limit problems. None of the 183 is GLM's: a run of main 7c0688951
  with kernel contract: the CUDA runner maps fp8_experts' image, so a run reaches its summary #1579 on another GB10 prints the same 183 lines over 1,327 launches (1,329 here), records of Qwen4's, Qwen3.5's
  and nvfp4_experts_tile's kernels with no 13.0 usage or another one.
• Tests run (Linux x86_64, at 2ff12c521): test-cuda-host 274/274 steps (4,103 of 4,112 tests, 9 skipped); zig build test 449/449 steps (1,872 of 1,890 tests, 18 skipped);
  test-kv-pool 30/30 tests, compile-cuda-tests 10/10 steps (the CUDA runner, which CI does not build),
  compile-config-server 7/7, test-roce-multi-host 40/40 tests, test-tp 33/33 steps,
  zig fmt --check, and tools/zig/lean_check.py with nothing new (424 against main's 424).
• Not run, and why: Metal (nothing here reaches it); CUDA GPUs other than GB10 (off there by default); a ring of four
  (a fourth node arrives in early November).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (The side
  stream only reads weights into L2; drafts on and off are both in the receipt.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (The
  token hashes agree across drafted and plain launches, on and off. The prompt cache was off and the four-stream runs
  were not hashed; the prefetch writes nothing, so every sum is main's.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No bits
  change: the side stream writes nothing, and every launch's tokens match.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's
  SUMMARY line is in the receipt. (GLM's l2pf and l2pace launches have nvcc 13.0 records; the CUDA lines are in the
  receipt.)
• Prompt processing is no slower than the last release. (It never runs in a prompt pass.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (CUDA on GB10 at
  TP=3 and TP=2, the host tests on Linux; the rest not run, as above.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (Each fails with
  its property taken out: take's budget, the table's lane order and its unknown buffer, the knobs' bounds, the
  fingerprint's l2pf, the records' measured usage and the GB10 default; they need no device.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Hi @Bizuayeu, a status update on #1580.

We read your three commits and the two kernels. They only prefetch into L2 and write nothing, so we found nothing
blocking. It waits on #1524, which it's stacked on, so please rebase once #1524 has landed. The run after that
needs two GB10s at once, and compares TF_GLM_L2PF=0 against unset on the same binary. Main doesn't know that
setting, so it can't be a main-against-head A/B. #1579 is on main now, so the kernel contract run will reach its
summary. If you can, a single-GB10 decode figure would help, since that's the default-on path no receipt covers yet.

[Bizuayeu]

If you can, a single-GB10 decode figure would help, since that's the default-on path no receipt covers yet.

(#1580 (comment))

GLM-5.3-Flash doesn't fit on one GB10: the XSFP4 checkpoint is 166 GB, and a GB10 has 128 GB shared between the host and the GPU, so it can't be served at TP=1 here. TP=2 and TP=3 are the GB10 setups that run it, and both are in the receipt. I'll rebase once #1524 lands.

[ashhart]

@Bizuayeu, #1524 landed at 00:01 GMT today, so this is clear to rebase onto main whenever suits you. Once it's pushed, we'll run TF_GLM_L2PF=0 against unset on our two GB10s at TP=2, since the 166 GB checkpoint can't run on one: tokens equal, decode at 1 and 2 streams, and cold prefill.

[Bizuayeu]

#1524 landed at 00:01 GMT today, so this is clear to rebase onto main whenever suits you.

(#1580 (comment))

Rebased onto main 24358c1e2, now that #1524 is in. #1524's commits drop out and this branch's three are unchanged (git range-diff) but for one line of context in contract_limits.zig. The receipts in the description are from before the rebase. On the new head 7b4099d12: test-cuda-host 268/270 steps (4,003 of 4,013 tests, 9 skipped), the one failure cuda_cost_identity's timing-dependent test, as on main; zig build test 445/445 steps; test-kv-pool 28/28, compile-cuda-tests 10/10, compile-config-server 7/7, test-roce-multi-host 40/40, test-tp 33/33, zig fmt --check, and tools/zig/lean_check.py with nothing new (425 against main's 425). It merges clean with main 94b90c9db.

[Bizuayeu]

zig-windows on 7b4099d12 was cancelled at the job's 60-minute limit, with nothing printed by zig build test after its first ten minutes and no test reported as failed. The other eight checks pass. The same three commits passed that job in 25 minutes on the previous head, and #1423 and main 94b90c9db passed it this morning, so I can't tell whether it's the runner or this branch. Could you re-run that job when you pick this up? I can't from a fork.

[Bizuayeu]

Rebased onto main ab4b5d8e6, which this branch conflicted with. The three commits are unchanged (git range-diff) but for context lines in cuda.zig and cuda_engine.zig from #1732 and registry.zig, and the landed_cuda line, which takes main's exl3_grouped_records term. The receipts in the description are from before the rebase. On the new head 2ff12c521: test-cuda-host 274/274 steps (4,103 of 4,112 tests, 9 skipped); zig build test 449/449 steps; test-kv-pool 30/30, compile-cuda-tests 10/10, compile-config-server 7/7, test-roce-multi-host 40/40, test-tp 33/33, zig fmt --check, and tools/zig/lean_check.py with nothing new (424 against main's 424). It merges clean with main 87726b034.