[jschmied]

Option C from #1554: a lane's prompt pass chooses its rows when it starts. It takes the large chunk when no other stream is decoding and the small one otherwise.

What this changes

• TF_FLASHNEXT_CHUNK_SOLO (opt-in) is the large chunk. It takes TF_FLASHNEXT_CHUNK to 16,384 rows; a value below TF_FLASHNEXT_CHUNK is refused at load. Unset, every pass takes TF_FLASHNEXT_CHUNK and nothing changes.
• step in cuda_lanes.zig picks the pass's rows from the other lanes as it starts. A lane whose prompt is in counts as decoding; one still filling its prompt does not. A pass is never cut short, so a request arriving during a large pass waits for it, as Flash Next CUDA: prompt chunk size against decode waits, how other engines handle it, and options #1554 describes.
• The engine's scratch, the drafts' rows and the TP=2 rank identity are sized on the larger value. The startup line reports it, and a log line names both chunks when they differ.
• The chunk parsers move from cuda_prompt.zig to a new cuda_chunk.zig (it was at 599 lines), with host tests for the parser and the choice.
• The recipe lists the variable and the table below. docs: Flash Next CUDA recipe, what TF_FLASHNEXT_CHUNK trades: faster prompts against longer decode waits #1553's text stays.

Receipts

GB10 (driver 615.71.09, nvcc 13.0), this branch at 63f1ee5 on main 4b287e3, nvidia/Qwen3.8-Flash-Next-FP4-Local at b9ce18a6, --parallel 3 --context 40960 --no-drafts --prompt-cache-gib 0, greedy. There were six fresh servers in the order unset, 8192, 4096, 4096, 8192, unset. Each cell is the median over both servers of a setting: 6 requests for first token, 4 runs for decode waits and 6 for arrivals.

	unset (1,536)	TF_FLASHNEXT_CHUNK_SOLO=4096	=8192
First token, one stream, 1,927 / 8,320 / 31,736 tokens	0.99 / 4.07 / 16.05 s	0.91 / 3.88 / 14.94 s (−8 / −5 / −7 %)	0.89 / 3.86 / 14.78 s (−10 / −5 / −8 %)
Longest wait between a decoding stream's tokens while an 8k prompt runs (median / max)	0.81 / 0.91 s	0.81 / 0.82 s	0.85 / 0.92 s
First token of a short request sent 2 s into a 31,736-token solo prompt (median / max)	0.36 / 0.37 s	1.91 / 2.28 s	1.94 / 2.27 s
That 31,736-token prompt's first token	16.81 s	16.90 s	16.67 s
Model and scratch at startup	71.39 GiB	73.64 GiB (+2.25)	77.25 GiB (+5.86)
MemAvailable 10 s after load (both servers)	44.1 / 46.0 GB	44.4 / 44.2 GB	38.0 / 40.4 GB

• Decode waits: two streams decode 400 tokens each while a third sends an 8,320-token prompt. With C on, that prompt's passes take 1,536 rows, so the waits stay at today's level. Every setting had one run near 0.91 s, the unset one too, so I read the 0.85 median at 8,192 as noise between starts.
• Arrival cost: this is what C adds. A solo pass lasts about 1.9 s at 4,096 rows and 3.8 s at 8,192, and that is the longest a request arriving during one waits before its own prompt runs. Once it decodes, the long prompt's passes drop to 1,536 rows, and the long prompt's own first token moves by under 1 %.
• 4,096 against 8,192: 4,096 took 80 to 90 % of the gain for 2.25 GiB instead of 5.86 GiB. Its arrival wait is also half as long in the worst case.
• Memory: the startup line's model figure includes the prompt scratch and is the stable number. MemAvailable moved by 2 GB between servers of the same setting on this shared box.
• Exactness: every reply was byte-equal to the first unset server's on all six servers, 23 texts a server. These were the 9 solo first-token requests, the decode-wait runs (2 × 400-token decoders plus the prompt) and the arrival runs (long plus short). The short request alone gave the same text as it did concurrently. In the arrival runs at C on, the long prompt changes chunk mid-prompt (8,192 or 4,096 rows, then 1,536).
• Default: C is slower in one row, the arrival, so it stays opt-in. The rest of the table is equal or faster.
• Tests: zig build test, test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon, git diff --check and a plain zig build native, all green. lean_check is 425 problems, the same as main, 0 new. Every touched file is at or under 600 lines.
• Not run: drafted serving, TP=2 and the MLX 4-bit checkpoint. The drafted path takes the same rows through prefillChunk, whose buffers are sized on the larger chunk.
• Overlap: the EXL3 Flash Next stack (EXL3 Flash Next on CUDA: one rank, offloaded experts, sm_120 gate (1/8) #1441–EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446, EXL3 Flash Next: host-resident K/V on the tier (6/8) #1522) edits the same step. The conflict is one adjacent line (warmAhead) and the moved parsers in cuda_prompt.zig; keeping both sides resolves it. Main's 7bf9711 (Python engine features to carry into the native engine #840), which landed while this ran, gives step a third argument. That is the one conflict with main, and it is also resolved by keeping both sides.
• Raw runs, server logs and scripts are kept locally under notes/tf1554c/ and can be uploaded on request.

For #1554; whether C becomes the default is the maintainers' call on these receipts.

[ashhart]

Thanks @jschmied, this is a clear write-up. Option C as an opt-in fits: the arrival cost is plain in your table, and anyone who serves one long prompt at a time can choose it.

We've rebased it onto main. One conflict: step gained a third parameter on main since you branched; we kept main's signature with your comment. We're running Flash Next FP4 on a GB10 with the variable unset, main against this, alternating fresh servers, to confirm the default path is unchanged in tokens and speed. If it is, it lands as is, with a changelog line crediting you.

[ashhart]

First result, @jschmied: with TF_FLASHNEXT_CHUNK_SOLO unset, Flash Next FP4 on a GB10 (main against this, alternating fresh servers) gives prompt tokens equal to main (0 of 20 rows differ), prefill -0.2% at 2k, decode -0.1%, and the stream sweep level at 1, 2, 4 and 8 streams (0.0% to +0.3%). So the default is unchanged, as you said. A second run on the MLX checkpoint is queued. It lands once our Mac gate is back, after an overnight Xcode update broke it.

[ashhart]

Second result, @jschmied: the MLX 4-bit Flash Next checkpoint on a GB10, TF_FLASHNEXT_CHUNK_SOLO unset, main against this. Tokens are equal, prefill is level at 2k and 8k (-0.3%, +0.8%), decode is level, and the stream sweep is +0.1 to +0.3% at 1 to 8 streams. One of the two runs of this PR read 64k prefill at about 1,340 tok/s, against 1,720 for both main runs, but the other run read 1,696 to 1,728, matching main cell for cell. So that's the box, not the code. With the FP4 run, the default path is unchanged on both checkpoints. It's ready, and lands once our Mac gate is back.

[ashhart]

Thank you @jschmied. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in 1.2.1, credited to you in the changelog and the release notes.