Repository navigation
Conversation
fn_hc_normalize ran one block a (row, stream): at one decode row four blocks scaled 10,240 values while thread 0 summed the stream's partials. fn_hc_normalize_cols takes a block a (row, stream, 256-column slice); each block sums the same partials in the same order, so every value's bits are fn_hc_normalize's, with forty blocks a decode row. The old kernel stays (the contract's baseline). Contract: replaces fn_hc_normalize bit for bit (gates 1-3; gate 4 states the bf16 sum it cannot show through the root), its control the new kernel with a doubled epsilon (red at gate 3). Served (FP4-Local, no drafts): 23.83 / 23.83 tok/s one stream (main 23.58 / 23.66), replies equal.
…up, eight groups a block fn_hc_dense_mix ran one 32-thread block a (row, 32-column group): 80 blocks of one warp a row at 2,560 dims. fn_hc_dense_mix8 runs the same per-value ops and the same in-order group sum with eight groups a 256-thread block, so its bytes are fn_hc_dense_mix's. Contract: fn_hc_dense_mix8 replaces fn_hc_dense_mix (8 shapes bit-equal, nonfinite rule stated, repeat); its bound adds the kernel's bf16 roundings of each term and the mean. Control: the same kernel with its inputs swapped, red at shapes. GB10, FP4-Local, 4,096-row prompt chunks, no drafts: replies identical to main on the 8k and 32k prompts; TTFT within run-to-run noise (8k 4.37-4.78 s vs 4.21-4.84 s, 32k 18.0 vs 17.9-18.8 s).
|
Thanks @jschmied for keeping both new launches row-local and for checking every output byte, the XSM group sums included, against the old kernels at gate 3. We are carrying both commits onto current main ourselves, kept as yours, so there is no need to drop the - Synthema, Ash's AI Assistant. |
… file, flashnext_hc.cu split, gate 4 for the mix's group sums - fn_hc_normalize_cols and fn_hc_dense_mix8's gate-5 records go into flashnext_cuda_records.zig, where main now keeps the Flash Next records; registry.zig stays as main's. - flashnext_hc.cu reached 601 lines with main's prompt-row read-out kernels: the dense normalize and mix kernels move to flashnext_hc_dense.cuh, included in place, so the image and its preprocessed source are unchanged. - contract_hc_mix.zig: a group-sums entry for fn_hc_dense_mix8's fp32 XSM sums, whose gate 4 groups hold 4096, 1, -4096 and zeros (1 in fp32, 0 with bf16 or fp16 sums), and a planted control, tf_ctl_hc_mix8_bf16, summing each group in bf16, which must go red at gate 4; a host test checks the groups' sums in every order.
|
Landed on main in 93a215b..35e46c0, with your records under your authorship plus the maintainer fix-ups. Thank you. Measured with the whole group (#1139, #1140, #1143, #1147, #1164, #1169, #1171 and #1170's Flash Next slice) against main on GB10s:
- Synthema, Ash's AI Assistant. |
What this changes
fn_hc_normalizeran one block a (row, stream): at a decode row four blocks of 256 threads scaled 10,240 values whilethread 0 of each summed the stream's ten square-sum partials.
fn_hc_normalize_colstakes a block a (row, stream,256-column slice); every block sums the same partials in the same order (
__fadd_rn, then the same IEEE division andsquare root), so every value's bits are
fn_hc_normalize's, with forty blocks a decode row.hcNormalizetakes itwhenever the width is a multiple of 256 (Flash Next's 2,560);
fn_hc_normalizestays as the contract's baseline.A second commit does the same for
fn_hc_dense_mix, one 32-thread block a (row, 32-column group):fn_hc_dense_mix8runs eight groups a 256-thread block, a warp each, with the same per-value ops and the same in-order group sum, so
its bytes are
fn_hc_dense_mix's.Flash Next's decode spends 9.9 ms a token in its hyper-connections (per-op-class cudaEvent timing), of which the
normalize was 0.82 ms (96 calls of ~8.5 us); the rest is the two BF16 projections and three more row kernels.
Receipt
maind1018b31plus this branch; one DGX Spark (GB10, sm_121, CUDA 13.0, nvcc 13.0.88);nvidia/Qwen3.8-Flash-Next-FP4-Local@b9ce18a6.main: three chats and three ~8k agent contexts byte-identical.--no-drafts, one stream, code / chat): 23.83 / 23.83 tok/s,main23.58 / 23.66 (+1.0 / +0.7 %).main6.09-6.96; with both commits at 4,096-row chunks4.37-4.78 s against 4.21-4.84 s, 32k 18.0 against 17.9-18.8 s; replies identical).
tf-cuda-test kernel-contracton the GB10: the new entry (zig/tests/cuda/contract_hc_normalize.zig)replaces
fn_hc_normalizebit for bit: gate 1 by the stated rule (NaN and infinite rows, a NaN stream sum, a zerosum), gate 2 five runs equal, gate 3 bit-equal at 6 shapes (1, 3 and 17 rows; 512 and 2,560 columns), gate 4: the
only sum is the stream's partials, so fp16 sums miss by 256x while a bf16 sum's error stays under the bf16 output
rounding after the root (the spec states
sums.bf16 = 0). Control: the new kernel with a doubled epsilon posing asthe replacement, red at gate 3. Gate 5: its record (32-40 registers, 16 static bytes, 256-thread bound).
fn_hc_dense_mix8(zig/tests/cuda/contract_hc_mix.zig) replacesfn_hc_dense_mixbit for bit: gate 1 by thestated rule, gate 2 five runs equal, gate 3 bit-equal at 8 shapes, gate 4 within its bound with the kernel's bf16
roundings of each term and the mean added (four bf16 terms a value:
sums.bf16 = 0, stated). Control: its inputsswapped, red at gate 3. Gate 5: 34 registers, 1,024 static bytes, 256-thread bound.
zig build test --summary all(built-Dnvcc): 211/211 steps, 2,959 of 2,973 tests passed, 14 skipped.lean_check.py+lean_new.pyagainstd1018b31): 0 new findings.Gotchas
fn_hc_dense_mix8) wasn't timed on its own. The decode numbers above are from the first commit, and the second showed no prompt gain beyond noise. It is bit-identical and contract-registered; I'll drop it if you prefer the smaller diff.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Measured with MTP drafts at--parallel 4: drafted == plain 8/8, concurrent == solo 8/8, and a two-turn prompt resumed from the cache (7,147 of 7,184 tokens restored) equals the same turn on a fresh server; each output equalsmain's.