[Bizuayeu]

Option C from #1554 on GLM, as asked on #1576: a prompt step takes two grid chunks in one pass while no other stream is decoding. Unset, it holds nothing extra and keeps the 2,048 grid, so warm turns and decoding streams are untouched.

On GB10 (GLM-5.3-Flash XSFP4, one binary, unset against TF_GLM_CHUNK_SOLO=4096, --parallel 4 --context 524288, TF_KV_PAGES=on, MTP drafts on):

	TP=2 unset	TP=2 =4096	TP=3 unset	TP=3 =4096
First token of a cold prompt, 8K / 32K / 128K (s)	6.08 / 23.94 / 98.16	5.72 / 22.22 / 91.33	5.00 / 19.61 / 80.26	4.64 / 17.99 / 73.45
First token of a warm turn, 10K history + 500 / 1,500 / 3,000 tokens (s)	1.40 / 2.06 / 2.87	1.40 / 2.05 / 2.83	1.14 / 1.67 / 2.35	1.14 / 1.68 / 2.31
First token of a warm turn, 40K history + 500 / 1,500 / 3,000 tokens (s)	1.25 / 2.03 / 3.05	1.26 / 2.04 / 3.03	1.02 / 1.67 / 2.51	1.02 / 1.66 / 2.48
Gap between a decoding stream's tokens beside a 32K prompt, median / longest (ms)	1,474 / 1,749	1,478 / 1,751	1,219 / 1,444	1,219 / 1,459
That stream's tokens during the prompt	17.0	16.5	16.0	16.5
First token of a short request sent 6 to 8 s into a 32K prompt, mean (lowest to highest) (s)	1.27 (0.88 to 1.65)	2.48 (1.35 to 3.47)	1.24 (0.70 to 1.67)	1.73 (1.07 to 2.67)
That prompt's first token (s)	23.54	23.27	19.32	18.99

The cold row is at --prompt-cache-gib 0, on Python source of 8,192, 32,768 and 131,072 tokens (bench_cache.py --require-cold, three rows a launch). A repeated filler line gains less, as on #1576. At about 7,820, 32,450 and 136,170 tokens, also at cache 0, it goes from 5.74 / 22.88 / 97.05 to 5.57 / 22.15 / 94.72 s at TP=2, and from 4.73 / 18.79 / 79.69 to 4.52 / 17.89 / 76.14 s at TP=3. The other rows have the prompt cache on: the default at TP=2, 2 GiB at TP=3. Each cell is the mean of two launches of a setting, but TP=2's cold row, which is one launch each. Three rows that waited on the heat guard are left out (below).

What this changes

• TF_GLM_CHUNK_SOLO=4096 (opt-in) on every rank. The value is rows, as in Flash Next's. Unset and 2048 are main's path and any other value is refused at load. 4,096 is the only wide size because a pass has to fit the index ring, which is 4,096 rows.
• Lanes.step asks Leader.chunk for a solo pass while no other lane is past its prompt pass, and prompt_solo.at allows it when the next grid point has at least two rows past it and no kept-state mark at it, and the pass rebases the rings where chunks one at a time would. The other ranks read the step from the part's length, as they read every step, and refuse a part that prompt_solo.allowed does not.
• The step stays at 2,048 rows with prompt logprobs or image rows. A drafter's steps take one chunk, so --drafter beside this is refused by name before the weights load, and MAX_CTX stays out of this PR. The grid, the rings and prompt_rows stay at 2,048, so a resume lands where main's does.
• The MTP head absorbs a solo pass 2,048 rows at a time (prompt_solo.mtpRows), because its ring holds a window more than a pass. That is the same rebases and the same state as chunks one at a time, and it takes cuda_engine.zig from 590 to 584 lines.
• Only the prompt buffers grow, sized at start. Unset, prompt_solo.bufferRows keeps them at 2,048 rows.
• Every rank has to agree, since rank 0's solo passes need the wider buffers on every rank, so Fingerprint carries solo. That takes layout 14, which glm cuda: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's sites #1423, glm cuda: TF_GLM_L2PF reads a decode window's next weights into L2 on a side stream, on by default on a GB10 #1580, glm cuda: TF_GLM_KV_HOST keeps the KV pages' latent rows in pinned host memory #1795 and glm cuda: TF_GLM_KDA_PAD packs the fused KDA input at TP=3 (opt-in, default off) #1545 also claim. Whichever lands later takes the next one.
• The startup line's "prompts in passes of up to" also feeds the kept-state grid, so it stays at 2,048 and a log line names both sizes: glm: prompt passes of 4096 rows, 2048 while another stream decodes.
• The recipe gets a ## Requests bullet and a ## Prompt chunk size section with the table above.

Receipts

• Environment: TensorFold 5c7eeaf33 on main 72cc3ea89, built with zig build native for sm_121 and served as one binary at both settings and both TPs. The commits on top are the recipe and the --drafter refusal, which drops a condition that was false in every run here. GLM-5.3-Flash XSFP4. TP=3 on three GB10 cabled in a ring of three QSFP cables with no switch, TP=2 on two GB10 joined by one QSFP cable. Launches alternate unset, set, set, unset, then again at --prompt-cache-gib 0 (TP=2: unset, set).
• Heat guard: no launch waited at TP=3. At TP=2 three of the six waited 2.0 s once each (heat_wait_s), one unset and two set, and the table leaves out the rows they fell in: one arrival run's long prompt at each setting, the short request beside the set one, and one of the three 128K rows. With them in, the 128K cell is 91.98 s, the short request 2.68 s (highest 4.05) and the long prompt 23.79 and 23.52 s.
• Warm turns and decoding streams: these stay at main's. Set, every warm turn resumed on the 2,048 grid (96 turns over both TPs, half of them off the 4,096 grid, none from nothing). In the gaps runs, three drafts-off streams decode while a 32K prompt fills, and with a stream decoding its passes take 2,048 rows. Decode alone stays too: at both settings gpu-chat-no-think reads 44.67 to 44.72 tok/s at TP=3 and 35.60 to 35.79 at TP=2, and fibonacci-raw 57.34 to 57.46 at TP=3, with one set launch at 56.20, and 43.48 to 43.62 at TP=2.
• Arrival cost: this is what the setting adds. A request arriving during a solo pass waits for it to end before its own prompt runs. A whole solo pass is about 2.3 s at TP=3 and 2.8 s at TP=2 (4,096 rows at the cold row's pace), against 1.2 and 1.5 s for a chunk. The table's are eight requests at four fixed points into the prompt (seven at TP=2 set, above), so they sample the wait and don't give its distribution. They are 31 tokens, about 0.15 and 0.2 s alone, and their prefill_seconds rises from 0.62 to 1.00 s at TP=3 and from 0.78 to 1.25 s at TP=2. As I read it, that is the end of the long prompt's pass, still on the GPU when their step begins, so it belongs to the same wait and isn't their own rows running slower. I haven't timed that split. Once the request decodes, the long prompt's passes drop to 2,048 rows, and its own first token still comes sooner than unset.
• Memory: the startup line's model goes from 63.34 to 64.90 GiB at TP=3 and from 90.59 to 92.51 GiB at TP=2, held for the whole run. The default prompt cache at TP=2 drops from 4.7 to 2.8 GiB, because the buffers come out of the same room. Not measured on this branch: the fixed 4,096 on glm cuda: prompt chunks of 4,096 rows, and the DFlash2 context buffers sized to match #1576 holds the same buffers, and at TP=3 with the 1M window it left room for 2 whole windows instead of 3 (kv_pool.windows, 38.0 against 39.55 GiB). --parallel 4 still started there.
• Exactness: glm-solo-check runs eight prompts on one engine, each a step at a time and solo where it can, with a mark at chunk 1 in one of them. The first token, the next window's logits and tokens, the rings' bases and every byte of the state equal their twins, on every rank at both TPs. With =4096, glm-lanes-concurrent passes at both TPs, and its streams of about 4,170 rows filled a step at a time equal their whole passes. glm-lanes-cover passes at TP=2 and fails only "copy drafts, sampled" at TP=3, as main's runner does; it fills whole passes, so it shows that path left alone. Twins of 8K, 32K and 64K prompts, greedy and sampled, each fresh, again and twice at once, give one hash a set across unset and =4096 (6 sets of 8 replies at each TP), and again with TF_GLM_KV_SHARE=16 beside =4096 at TP=3. The three cold prompts hash the same in every launch. At TP=2 with =4096 and --prompt-cache-gib 0, the 8K, 32K and 64K prompts, greedy and sampled, each filled cold and alone, give one hash over 256 tokens between "draft": false and drafted (6 of 6 pairs, 1,200 rows drafted).
• Tests (Linux x86_64, at a27444ebb): test-cuda-host 276/276 steps (4,183 of 4,192 tests, 9 skipped), zig build test 451/451 steps (1,879 of 1,897 tests, 18 skipped), test-kv-pool 33/33, compile-cuda-tests 10/10 (the CUDA runner, which CI does not build), compile-config-server 7/7, zig fmt --check, and tools/zig/lean_check.py on the merge with main 57186386d (424 against main's 424, 0 new). Every touched file is at or under 600 lines on that merge (cuda_lanes.zig is at 600 on main and stays there).
• Not run: Metal (nothing here reaches it), CUDA GPUs other than GB10, and a start with a real DFlash2 drafter (the refusal is in the host tests). The 1M window was not served with this on.
• Overlap: glm cuda: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's sites #1423's pair and this step both reach the other ranks as a part of two chunks, so one server cannot mix them. With this set, the pair should stand aside and hold no twin. This step's conditions include the pair's, so nothing is lost there, and the 4,096-row pair (GLM-5.3-Flash on CUDA: prompt chunks of 4,096 rows cut cold first-token time 13% at 32K at TP=3 (and stack with #1423) #1570's stacked form) is not in this PR. Whichever lands second carries that rule and folds the two othersDecode. Merging this with glm cuda: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's sites #1423 today conflicts in 8 files, and with glm cuda: TF_GLM_L2PF reads a decode window's next weights into L2 on a side stream, on by default on a GB10 #1580 in 2, keeping both sides. glm cuda: TF_GLM_KV_HOST keeps the KV pages' latent rows in pinned host memory #1795 edits the same lines of cuda_ranks.zig, its test, cuda_native.zig and the recipe's ## Requests list, and glm cuda: TF_GLM_KDA_PAD packs the fused KDA input at TP=3 (opt-in, default off) #1545 the field list in that test; keeping both sides and taking the next number resolves those. Against main 57186386d this merges clean.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (A solo pass is one lane step of two chunks, and it runs with the MTP drafts on.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (The twins above cover resumed and concurrent, and the six pairs after them cover drafted against "draft": false on a solo pass, at TP=2.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No sum changes; glm-solo-check compares the bytes.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (No kernel changes.)
• Prompt processing is no slower than the last release. (Unset every step is main's; set, the table.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (CUDA on GB10 at TP=2 and TP=3, the host tests on Linux; the rest not run, as above.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (The host tests read takesTwo, ringsAgree, bufferRows, othersDecode, refuse and Settings.solo, which main does not have; glm-solo-check needs a GPU and a checkpoint.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thanks @Bizuayeu, this is the shape we hoped for. Unset holds nothing extra and keeps the 2,048 grid, the other ranks
check the same rule rank 0 uses, and the recipe states both costs: the wait for a request that arrives mid-pass and
the 1.6 to 1.9 GiB of prompt buffers. glm-solo-check comparing every state byte stepped against solo is a good
test to have. We'll land it after our runs on our GB10s. One note: #1795 also takes Fingerprint layout 14, so
whichever lands second moves to 15. One small optional addition: with --drafter set, the buffers are still
sized for 4,096 rows but no pass uses them, so a load-time warning (or a refusal by name) would save someone the
memory.

[Bizuayeu]

One small optional addition: with --drafter set, the buffers are still sized for 4,096 rows but no pass uses them, so a load-time warning (or a refusal by name) would save someone the memory.

(comment)

Done in a27444ebb: TF_GLM_CHUNK_SOLO=4096 beside --drafter is now refused by name before the weights load, as TF_GLM_KV_SHARE without pages is, so nothing is held. That left the step's own drafter conditions with nothing to catch, so they're gone. Neither was ever true in the receipts' runs, so those stand. I haven't started it with a real drafter; the refusal is in the host tests. On layout 14, noted: if #1795 lands first I'll move this to 15.

[ashhart]

Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.