Repository navigation
Nemotron 3.5 Transcribe: audio input, the 65,536-token window, an hour of audio, MTP drafts - #1632
BobClawblaw wants to merge 10 commits into
Conversation
06b023c to
ba10f75
Compare
ba10f75 to
3ae00f5
Compare
3ae00f5 to
a18905c
Compare
|
Rebased onto d377fcc (with #1630 and #1631 under it). Two changes for lean, both pure moves:
Checked on the pushed head: fmt, lean (0 new) and On a GB10 (first rebase, d887be514 against the pre-rebase head 3ae00f5):
On an RTX 5090 (dd7cc1a06, with #1769's brace fix on top, which main needs to build CUDA at all): |
…nt blocks (relu^2 up, fp32 or bf16 down)
…and bf16 down, gates 1-5 with two controls
…m) from its bf16 weights, audio rows spliced after the embeds The checkpoint's llm.* tensors load as published: NeMo's layer names, fp32 A_log/D/dt_bias, the stacked [E, in, out] experts on the bf16 grouped format with the shared expert as experts E and E + 1, TE's empty _extra_state skipped. The audio encoder and CTC aligner tensors are left for the audio path. A prompt pass's audio rows replace the <|audio|> positions' embeddings, as image rows do.
… ASR encoder, streaming Sortformer, fusion), audio-check NeMo's vLLM SALM path as ParallelExpertEncoder's online inference runs it: the log-mel frontend on the host, the ASR TransformerEncoder over context-extended 500-frame windows, the Sortformer over the same windows with its speaker cache and FIFO, thresholded speakers through the sinusoid kernel, and the projection to the language model's width. nemo_audio.cu adds q/k LayerNorm with partial RoPE, attention over a window's valid keys and the Sortformer head's glue; the rest runs on the vision towers' kernels. audio-check writes every stage under the reference's names.
…r FIFO and pooling) run under test-cuda-host
The Bolt load-recovery tests build an Engine by hand and null each optional kernel set; bf16x was left undefined, so deinit unloaded a garbage module (seven tests crashed on Linux and Windows CI).
…o_url WAV parts), encoded at each request's prompt pass, rows keyed by clip in the prompt cache
… a request (160 MiB chat bodies when the engine takes audio), MTP drafts with the clip's rows in the head
… audio input), its index row and the API's audio parts
a18905c to
540e914
Compare
|
Rebased onto fbbc20d. Main's YaRN work added Checked on the pushed head: fmt, lean (0 new) and |
What this changes
Closes #1549. The server takes audio for Nemotron 3.5 Transcribe: OpenAI
input_audioand vLLMaudio_urldata-URLparts with 16 kHz mono WAV in user messages, encoded at each request's prompt pass, a clip's rows keyed in the prompt
cache. The model gets its trained 65,536-token window (config.json's 262,144 is the backbone's), up to an hour of audio
a request (160 MiB chat bodies only when the engine takes audio), and MTP drafts with the clip's rows in the head.
Recipe:
docs/recipes/nemotron-3.5-transcribe.md. Stacked on #1631.Receipt
--context 65536, greedy, thinking off.--no-drafts--no-draftson all three;--prompt-cache-gib 0gives the same repliesas the default on resumed clips, seeded sampling, speaker tags, translation, language id, audio Q&A, thinking, two
turns; four long clips at once equal their solo runs; a repeated clip's first token 0.381 → 0.067 s.
SUMMARY 452 kernels: 452 pass gates 1-4, 0 fail, 6 known findings; 325 controls: 325 red as planted, 0 missed; 321 limit problems(the stack on main d33dcfc; none of the limit problems is in this stack's launches). The stack then rebased onto 3f0a715 without conflicts; the three sets,zig build testand the Transcribe clips were re-run there.Overlap
atc overlaplists #1445 and #1446 as conflicting, in the shared registration lists (registry.zig,cuda_kernels.zig,kernels.zig,kernel-contract.md,tests/cuda/main.zig); whichever lands second rebases.