[jschmied]

What this changes

Docs only (docs/recipes/flash-next-cuda.md):

• TF_FLASHNEXT_CHUNK is now listed under Run.
• A new section, Prompt chunk size, explains the trade-off from the code. A prompt of N rows runs in ceil(N / chunk) passes, and every pass reads the weights of each expert its rows route to. So fewer, larger passes make prompts faster, but a stream that is decoding waits up to the whole pass for its next token, and the prompt scratch grows with the chunk.

Receipt

• Code-backed claims: the range (1 to 16,384, refused before load), the 1,536 default, one chunk per prompt pass, 1.84 MB of gate and up weights an expert, 512 experts, the scratch growing with the chunk, and the 16,384-row paging note in docs/known-issues.md.
• Measured numbers: none. The first version had a timing table, an nsys comparison and an exactness line; their raw run files are gone, so they are cut. They can come back with a full receipt if the chunk default changes (Flash Next CUDA: prompt chunk size against decode waits, how other engines handle it, and options #1554).
• Tests run: none needed for a docs change; lean_check 425, the same as main.

[ashhart]

Thanks @jschmied. We checked every claim the code can back, and they hold: the range (1 to 16,384, refused before anything loads), the 1,536 default, the startup line, one chunk per prompt step, the 1.84 MB of gate and up weights per expert, 512 experts, scratch growing with the chunk, and the 16,384-row paging note in known issues. Two changes before we land it:

1. docs/recipes/flash-next-cuda.md:62-63: "a pass runs to its end before other streams decode their next token" reads as a guarantee about scheduling. It's really a cost. Suggested wording: "a pass runs to its end, so a stream decoding during a pass waits up to the whole pass for its next token."
2. The measured numbers (:205-223): the timing table, the nsys paragraph (1,052 to 914 ms), "about 350 of the 512 experts" and the 64-token exactness line can't be checked from the PR. Could you attach or link the receipt: the requests, the three runs per setting, the wait and memory numbers, and the token hashes? Then they stand as measured. If you'd rather not, cut those lines and keep the code-backed explanation, which is useful on its own. If the pass count stays, give it as arithmetic (1,922 rows is two passes at 1,536 and one at 8,192), since cache marks can change the measured count.

[jschmied]

Thanks @ashhart. Both done in bc5e0d8:

1. Lines 62-63 now use your wording: a pass runs to its end, so a stream decoding during a pass waits up to the whole pass for its next token.
2. I cut the measured numbers: the timing table, the nsys paragraph, "about 350 of the 512 experts" and the 64-token line. Their raw run files are gone, so I can't attach a receipt that stands on its own. The section now keeps the code-backed explanation and gives the pass count as arithmetic (a 1,922-row prompt is two passes at 1,536 and one at 8,192). I updated the PR description to match. If the default changes under Flash Next CUDA: prompt chunk size against decode waits, how other engines handle it, and options #1554, the numbers can come back with a full receipt.

[ashhart]

Thank you @jschmied. This is in TensorFold 1.2.0, released today: https://github.com/ashhart/TensorFold/releases/tag/v1.2.0

It landed with your commits under your own name, so you're listed as a TensorFold contributor, and it's credited to you in the release notes and the changelog.