Repository navigation
Nemotron 3.5 Transcribe on CUDA: the language model and the audio encoder - #1631
BobClawblaw wants to merge 10 commits into
Conversation
|
CI on Linux job · Windows job. The stacked #1632 also has the same seven crashes on Linux at This is from the CI logs; I have not reproduced it locally or established whether the fault is in the test setup or the cleanup path. |
|
Thanks for this, @BobClawblaw. It reads well: everything new sits behind the nemo_speechlm config, and the shared Nemotron loader changes keep the old tensor names for every other checkpoint. It rebases cleanly onto #1630, so we'll land it once #1628, #1629 and #1630 are in. Two small things before then, if you have time:
We also noticed that |
|
Thanks @cwschroeder and @ashhart. Pushed cb83265:
#1632 carries the same crash and will be rebased onto this. The drafted == |
|
Thanks @ashhart.
And yes, |
cb83265 to
bc2fbea
Compare
|
Correction to my comment above: cb83265 was replaced on this branch by bc2fbea (which carries the nemo_audio test step instead). The |
…n NVFP4 block holding one gets a NaN scale (max_nan, not fmaxf)
…e shared expert's slot, block-row and tiled, fp32 and bf16 out), gates 1-5 with two controls
…nt blocks (relu^2 up, fp32 or bf16 down)
…and bf16 down, gates 1-5 with two controls
…m) from its bf16 weights, audio rows spliced after the embeds The checkpoint's llm.* tensors load as published: NeMo's layer names, fp32 A_log/D/dt_bias, the stacked [E, in, out] experts on the bf16 grouped format with the shared expert as experts E and E + 1, TE's empty _extra_state skipped. The audio encoder and CTC aligner tensors are left for the audio path. A prompt pass's audio rows replace the <|audio|> positions' embeddings, as image rows do.
… ASR encoder, streaming Sortformer, fusion), audio-check NeMo's vLLM SALM path as ParallelExpertEncoder's online inference runs it: the log-mel frontend on the host, the ASR TransformerEncoder over context-extended 500-frame windows, the Sortformer over the same windows with its speaker cache and FIFO, thresholded speakers through the sinusoid kernel, and the projection to the language model's width. nemo_audio.cu adds q/k LayerNorm with partial RoPE, attention over a window's valid keys and the Sortformer head's glue; the rest runs on the vision towers' kernels. audio-check writes every stage under the reference's names.
…r FIFO and pooling) run under test-cuda-host
The Bolt load-recovery tests build an Engine by hand and null each optional kernel set; bf16x was left undefined, so deinit unloaded a garbage module (seven tests crashed on Linux and Windows CI).
d5d067e to
64098fe
Compare
What this changes
Nemotron 3.5 Transcribe (
nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B@c421d6acdb436edd7ae00565f6414631fb9bd56d,the bf16 release as published) on CUDA: its language model on the Nemotron family (NeMo's names,
cuda_salm.zig), andits audio encoder (
families/nemo_audio: host log-mel, the 32-layer ASR encoder, the streaming Sortformer and theirfusion,
nemo_audio.cu), withtensorfold audio-checkwriting every encoder stage under NeMo's names. Exact bf16caches; no conversion. Part of #1549; stacked on #1630.
Receipt
fifth writes "at 'em" for "at em", a near tie that bf16 logits settle the other way.
features identical.
SUMMARY nemo_audio: 7 kernels, 7 pass gates 1-4, 0 fail; 5 controls: 5 red as planted, 0 missedSUMMARY 452 kernels: 452 pass gates 1-4, 0 fail, 6 known findings; 325 controls: 325 red as planted, 0 missed; 321 limit problems(the stack on main d33dcfc; none of the limit problems is in this stack's launches). The stack then rebased onto 3f0a715 without conflicts; the three sets,zig build testand the Transcribe clips were re-run there.nemotron/config.zigreadsnemo_speechlm; other Nemotron checkpoints load as before (the A/B inrelu^2 keeps a NaN: grouped experts, ck experts and Nemotron's pocket #1628 covers Lightning, Bolt and Super VL on this stack).
Overlap
atc overlaplists #1445 and #1446 as conflicting, in the shared registration lists (registry.zig,cuda_kernels.zig,kernels.zig,kernel-contract.md,tests/cuda/main.zig); whichever lands second rebases.