Repository navigation
Conversation
…d lerp residual Dory's residual stream lives on the unit sphere, so its rows need two kernels norms.cu did not have, both a block a row (256 threads, 16-byte loads, rows to 8,192 in 64s), so a row's bits never depend on the row count: - tf_l2_rows: y = bf16(s / sqrt(sum(s^2) + eps)), s = x, or bf16(x + r) (a loop's entry, justnorm(state + input_h); with r null, a loop's exit). - tf_slerp_rows: dory.py's slerp_norm, h = bf16(l2(a + |alpha scale| (b - a))) with a = bf16(l2(res)) and b = bf16(l2(out)), the lerp in fp32 steps. Norms.l2 and Norms.slerp launch them; no existing kernel or launch changes. Both are registered with the kernel contract. Their check compares every output with float64's value at the bf16 rounding points as a range over the fp32 run's spread (core/l2_ref.zig); three controls each drop one rounding point (the sum before the norm, the slerp's a, its b) and go red at gate 3. Gate 4's integer rows make an fp16 sum overflow to a zero output (85x and 51x the bound); a bf16 sum's scale cancels in the norm, so that margin is stated, not proven, as for the RMS tails. Launch records from nvcc 13.0's sm_80/86/89/120/121 images.
…2 norm and q's scale tf_qk_rows_l2 takes dory.py's order: rotate-half RoPE on the projection's bf16 rows (rounded to bf16), then each head's L2 norm with eps on the sum, then q times an fp32 scale vector ((sqk f)^2, formed on the host as torch forms it) after the normalized row's rounding; k is its normalized row, v as is, and k and v go into the caches as tf_qk_rows writes them. Shape.l2 and Shape.q_scale select it; every other call keeps its kernel. It is registered with the kernel contract: every output against float64's range at its rounding points (cosf's ulps on the fp32 angle pos * inv, the inverse's fp32 spread), at head dim 256, 16 query heads over 4, a third of the rows past position 199,800. Two controls each drop one rounding point (the rotated head, q's normalized head) and go red at gate 3. Gate 4's integer heads (no rotation) overflow an fp16 sum to a zero output. core/l2_ref.zig's intervals now keep infinite ends and widen by fp32's least subnormal step, so a planted infinity leaves only the outputs float64 makes non-finite.
…then L2-normed Two activation rows for Dory, a block a row, 16-byte loads, fp32 scale vectors formed on the host as torch forms them: - tf_gate_l2_rows: out = bf16(l2(bf16(o * sigmoid(g * sg)))) over all heads of a row (dory.py's output gate, then justnorm over num_heads * head_dim); - tf_swiglu_l2_rows: out = bf16(l2(bf16(silu(bf16(g * sug)) * bf16(u * suu)))) (its MLP). sigmoid and silu are torch's fp32 forms, 1 / (1 + expf(-x)) and x / (1 + expf(-x)) with IEEE division. ActRows.gateL2 and ActRows.swigluL2 launch them; no existing launch changes. Both are registered with the kernel contract: every output against float64's range at its rounding points (expf's ulps, fp32's exp range, the inverse's spread) at Dory's widths, 4,096 and 7,168; four controls each drop one rounding point (the gated row, the scaled gate, the scaled up, the product) and go red at gate 3. Gate 4: the gate's rows are halves of integers (sigmoid(0) = 1/2), exact in fp32; SwiGLU's squares are no integers, so its fp32 sums are checked against the shapes gate's bound, and an fp16 sum overflows either way.
Dory's logits are the head's bf16 output times logit_scale / std per vocabulary column (dory.py: logits.float() * (logit_scale.float() / initializer_range)). tf_dense_rows_scaled_* are the same tiles with that epilogue, out = fp32(bf16(x . w[n])) * scale[n] in one fp32 product; DenseRows.rowsScaled launches them. tf_dense_rows_* keep their code: their records from nvcc 13.0 are unchanged. Registered with the kernel contract as replacements: on the same inputs every output is bit-equal to its unscaled tile's fp32 outputs, rounded to bf16 and scaled on the host, at every served shape and edge (4,103 columns, 2,560 inputs, the row counts the wrapper sends each tile); gates 1, 2 and 4 against float64 of the same steps, with a NaN and an infinite scale planted.
…ry (host only) Nemotron-Labs Dory 1.0 is a looped nGPT stack: input sublayers once, the recurrent block n_recurrent_loops times, output sublayers once, attention and MLP sublayers alternating. - config.zig parses config.json and generation_config.json's end tokens: the block counts, sliding/full/mlp kinds, both RoPE thetas, the nGPT constants and their fp32 factors, the loop count (and withLoops, 1 to 4, for a load option). It refuses last_loop KV (chunk- dependent by construction), quantizations other than ModelOpt W4A16_NVFP4 at group 16 or plain bf16, scaled or partial RoPE, tied heads, biases and shapes the kernels do not take. - names.zig maps every tensor of the NVFP4 (or bf16) checkpoint with dtype and shape checks and refuses leftovers; the pinned checkpoint (TF_DORY_DIR) maps its 1,083 tensors. - schedule.zig gives the virtual sublayers for L loops with a per_loop cache per attention step: 8 + 20 L + 8 caches, matching llama.cpp #30306's physical[] and cache_layer[] and vLLM #60149's loop * layers + layer indices. - memory.zig counts weight bytes as stored (equal to the pinned file's tensor data) and KV bytes by loops, context and sliding ring: 36, 56 and 96 KiB a token at 1, 2 and 4 loops.
ref.zig runs Dory's forward in float64 over prompt chunks with a stream's KV caches, following vLLM #60149's dory.py (ce9dd0ed): the nGPT slerp residual, q and k L2-normed after RoPE with q scaled by (sqk f)^2, the scaled sigmoid output gate and the row L2 over all heads, the scaled SwiGLU and its row L2, the loop entry l2(state + input_h) and exit l2(h), and the logit scale. Options give dory.py's bf16 rounding points with torch's fp32 scale vectors, per-step taps, and last_loop KV. ref_test.zig checks, on a tiny random config (2 + 4 + 2 sublayers, window 3): - per_loop: one chunk, several chunkings and one-token decode give the same logits to the bit, at 1 to 4 loops, in float64 and at the bf16 points; - last_loop: equal to per_loop at one loop and within one chunk, but at two loops the rows after a cut differ, the chunk dependence that keeps it out of this family; - every logit within 1e-12 of dory.py itself run in float64 (its fp32 casts as float64) on the same weights at 1, 2 and 4 loops (ref_fixture.zig; the oracle is a dev-time script, not part of the tree).
nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8 and -BF16 next to the NVFP4 map. - config.zig: Quant gains mxfp8. quantOf takes compressed-tensors' mxfp8-quantized (8-bit float codes, group 32) besides ModelOpt's W4A16_NVFP4 and plain bf16, and refuses other compressed-tensors groups by name. - names.zig: an MXFP8 projection is its E4M3 codes [n, k] and UE8M0 scales u8 [n, k/32]; the tiny checkpoint test runs all three quants, and the pinned tests (TF_DORY_DIR, TF_DORY_MXFP8_DIR, TF_DORY_BF16_DIR) share one body. - memory.zig: the published MXFP8 and BF16 files' tensor data (4,750,663,680 and 7,950,458,880 bytes) equal weightBytes; parameters are the same for every quant.
…dows bit-equal
cuda_weights.zig loads the ModelOpt W4A16 NVFP4 checkpoint: every projection packed on the
host for the lane matmul with its own weight_scale_2 (q, k, v and g launch separately), the
nGPT scale vectors formed in fp32 as torch forms them ((sqk f)^2, (s_gate f)^2, suv
sqrt(hidden), logit_scale / std), embeddings and head bf16.
cuda_model.zig runs the sublayer schedule over per_loop KV caches (8 + 20 L + 8: full
caches over the context, sliding ones over a ring of a chunk and the window): embed, the
loop entry and exit L2 rows, the projections (prompt GEMM for prompt chunks, the lane for
decode rows), tf_qk_rows_l2 into the cache, window attention at head dim 256 (window 1024,
or none over the linear cache), the gate and SwiGLU L2 rows, the slerp residual, and the
scaled head. Prompt chunks to 2,048 rows, decode windows to 16.
tf-cuda-test dory MODEL LOOPS N TEXT, on nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4
(41f02335), GB10:
- greedy text is sane at 1, 2 and 4 loops ("The capital of France is" -> " Paris. The
capital of the United States is Washington, D.C. ...");
- drafted == plain: verify windows of 2, 3, 5, 8 and 16 rows copied from the plain run give
every row's logits bit-equal to the plain one-row decode (35 and 54 windows a run);
- a prompt cut in two chunks gives the last row's logits bit-equal to one chunk's, and the
prompt fed as decode rows gives the same first token.
Eager decode, no graphs: 14.7, 20.8 and 33.7 ms a token at 1, 2 and 4 loops.
… calls under host tests
…e slots a stream, graphs a row count The CUDA forward takes rounds of up to 128 rows over every stream's slot (per_loop caches [slots, len, HK, D], 8 + 20 L + 8 of them, sliding ones a 3,072-key ring), a lanes backend plugs it into lane_host with copy drafts, keyed draws and fp32 greedy picks, and decode forwards run as CUDA graphs captured per row count (TENSORFOLD_DORY_GRAPHS=0: eager). --loops (1 to 4, default the checkpoint's) is a load option of the server and of `tensorfold run`; other families refuse it. Admission counts each stream's caches by loop count and context. Kept prompt states copy every cache, the rings as rings, under a key of loop count and KV mode. Gates on the NVFP4 checkpoint at L=1, 2 and 4: eager == graphs (tokens, and a round's fp32 logits bit-equal), copy drafts == plain, 2/4/8 mixed streams == each alone.
…l the fresh pass tf-cuda-test dory-states MODEL LOOPS TEXT N keeps a state at 8000, 8192 and 32001 (off and on the chunk grid), wipes every cache to bf16 NaN, resumes a 2,600-row longer prompt (past a chunk and the 1,024 window) and checks greedy and sampled replies against a fresh pass. Green at L=1, 2 and 4.
7 of 9 tasks
This was referenced Oct 25, 2026
Owner
|
Landed on main in 6528fbe..81b476d: the whole Dory stack (#1256-#1261), with your records under your authorship plus five maintainer children. The children refuse host-only formats and refund failed upload ownership, pin release launch arguments, keep the unmeasured Dory launches pending in the contract, and own queued host inputs through stream completion. Every existing family file, contract entry and recorded launch is byte-identical. Thank you, Jürgen. On a GB10 with nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4 @ 41f02335, served at 2 loops with CUDA graphs:
- Synthema, Ash's AI Assistant. |
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Steps 6 and 7 of the Dory plan (#1233): the CUDA forward from #1260 served on
tensorfold-native, and kept prompt states. Three commits:server: Dory's chat template, end tokens, think split and qwen3_coder calls under host testsdory: served on tensorfold-native and tensorfold run at --loops, cache slots a stream, graphs a row countdory: kept prompt states resumed at 8k and 32k into wiped caches equal the fresh passThis PR closes nothing. The issue is #1233.
Stacking. Stacked on #1260 (family), on #1259 ... #1256; the #1245 contract include fix is #1256's base commit. Retarget as the earlier PRs land.
What it does
chat_template.jinjarenders byte-equal to Jinja2 in 5 cases, theenable_thinkingswitch works,<think>splits intoreasoning_content,<tool_call><function=is parsed by markup detection (qwen3_coder), and the end tokens are [2, 11]. These are host tests. Not reachable from a request: the template'smedium_effortandtruncate_history_thinkingvariables.[slots, len, HK, D], 8 + 20 L + 8 of them, sliding ones a 3,072-key ring). A lanes backend plugs intolane_hostwith copy drafts, keyed draws and fp32 greedy picks. Decode forwards run as CUDA graphs captured per row count (TENSORFOLD_DORY_GRAPHS=0: eager). Admission counts each stream's caches by loop count and context, in the room the weights leave.--loops(1 to 4, default the checkpoint's own, 2) is a load option of the server and oftensorfold run. Other families refuse it by name.Gates
Pinned NVFP4 checkpoint, GB10, L=1, 2 and 4, 48 tokens, 8 mixed prompts of 90 to 3,100 tokens (past the window and a chunk; 4 greedy, 4 sampled).
State sizes at L=2: 605 MiB at 8,000 tokens, 1.9 GiB at 32,001; at L=4, 3.3 GiB at 32,001.
Served smoke
tensorfold-native serve --loops 2(and 4), log "Dory at 2 loops (per_loop KV): 56 caches a stream, 8 streams, CUDA graphs on":reasoning_content;enable_thinking: falseworks;get_weather {"city":"Paris"}withfinish_reason: tool_calls;[DONE]; 4 parallel requests answer sensibly; a client cancel is followed by a normal answer.tensorfold run --loops 2: 20.38 ms a token with graphs, 20.60 eager, same token hash;--loops 4: 32.87 ms a token.Throughput
Decode only, 200-token prompts, 128 tokens, greedy, no drafts, GB10, NVFP4 (
tf-cuda-test dory-bench). Cells are tok/s together / tok/s a stream / ms a round.A single stream reaches about 74 / 71 / 69% of the bandwidth estimate (87 / 62 / 39 tok/s at L=1/2/4). Graphs barely help (20.4 against 20.6 ms at L=2), so the time is in the kernels, not the launches; the next lever is kernel-side.
Not in this PR
GPU lanes beyond copy drafts (head drafts), MXFP8/BF16 on the GPU, and paged KV on
kv_pool.Receipt
07742e446; NVIDIA GB10 (sm_121), CUDA 13.0 nvcc, Zig 0.17.0. The 27B vLLM server was not running (checkedfree -g).zig build test -Dnvcc=...: exit 0 at this branch head (zig build test -Dnvcc=/usr/local/cuda/bin/nvcc, the full step).main.mainplus the include fix,SUMMARY 280 kernels: 278 pass gates 1-4, 2 fail, 2 known findings; 53 controls: 52 red as planted, 1 missed; 1566 limit problems. On this stack's top:SUMMARY 290 kernels: 288 pass gates 1-4, 2 fail, 2 known; 62 controls: 61 red, 1 missed; 1566 limit problems. That is 10 Dory kernels and 9 Dory controls added, all passing or red; the 2 failures, the 1 missed control and the 1,566 limit problems are the same onmain(qwen35 decode attention,tf_q35_gdn_slotsrecords of 0). Two things onmainkeep the unmodified run from completing:w4a8_tests.zigdoes not compile againstw4a8.zig'spromptsignature, and the w4a8 kernels lose the CUDA context on their planted non-finite inputs. Both runs above skipped the w4a8 kernels and patched that test call locally, identically on both trees; neither patch is in this PR. All Dory launches pass the live limits check.tf-cuda-test dory ... 2 32 "The capital of France is"gives the same text, drafted == plain over 35 windows, the 3-of-6 chunk cut bit-equal; a served smoke at--loops 2("Dory at 2 loops (per_loop KV): 56 caches a stream, 8 streams, CUDA graphs on") answers "Paris", parsesget_weather {"city":"Paris"}withfinish_reason: tool_calls, and streams to[DONE]. The gate and throughput tables above were measured before the last rebase; the kernels are unchanged by it.Choices ashhart can change
These are the unanswered questions of #1233, and each follows the default noted:
--loopsas a load option, default the checkpoint's 2. A per-request count (for example viareasoning_effort) would cost max(L) per shared round and a graph set per L.per_looponly;last_loopis refused at config parse.kv_poolis a later step.