[jschmied]

Part of #1491 (CUDA side). Stacked on #1492 (LFM2.5 on CUDA, itself on #1377) and on BlivionIaG's LFM3 admission 8192e5437 (cherry-picked here with his authorship). Only the last four commits are new. Draft until those land.

What

LFM3-1.7B text served natively on CUDA from LiquidAI/LFM3-1.7B-NVFP4 (ModelOpt NVFP4), weight-only: the FP4 weights are dequantized, and activations stay bf16. MLX affine and FP8 come next on the same family; VL after that.

• Admission (b70b95a53): the ModelOpt NVFP4 scheme in Blivion's lfm3/config.zig / weights.zig, with the export's exact exclusion list: only the FFN (w1, w3, w2) is FP4; conv, attention, gate, embedding and head are bf16. The real header checks out at 502 tensors; the MLX 4-bit layouts are unchanged.
• Kernels (b5ed2eb69): the 7-tap instances of lfm2_moe: serve LFM2.5-24B-A2B on CUDA (plain serve, shared kernels; stacked on #1377) #1492's liquid_conv.cuh, tf_liquid_conv_7 / tf_liquid_commit_7, under the kernel contract with planted controls and records for nvcc 13.0 and 13.2.
• Prompt attention at head dim 128, 4 query heads per KV head (6040caa99): a new instance of prefill_attention's template.
• Family (08b338c7c):
  • FFN: one FP4 lane weight for w1|w3 on TF's NVFP4 linears (prompt16 for prompt rows).
  • Projections: q|k|v|gate as one bf16 lane weight.
  • Head: a bf16 copy of the tied embedding.
  • Full-attention layers (6): pool pages, partial RoPE through qk_rows' frequency table, Nemotron's head-128 chunk/merge kernels for decode.
  • Sliding layers (5, window 256): a per-slot ring on the shared window-attention kernels.
  • Gate and norms: the per-head output gate is act_rows.headGate; the sandwich norms are the shared norms.
  • Serving: --parallel N and the prompt cache (conv states, the rings and the pages through the cut, copied).

New kernels: none beyond the 7-tap instances. 25 files, +2,197 / −35 on top of the admission commit.

Verification (GB10, sm_121, real weights)

• float64 reference (CPU, NVFP4 dequantized exactly; it agrees 829/829 with the official modeling_lfm3.py run in float32):
  • Greedy agrees on 825 of 829 prompt positions and 127 of 128 reply tokens. The misses have float64 margins of 0.003–0.153.
  • With the engine's bf16 rounding points it is 827/829 and 126/128; the remaining misses are exact bf16 ties or 0.125 apart.
• Served:
  • drafted == plain 8/8 and concurrent == solo 8/8;
  • prompts of 300–2,960 tokens crossing the window several times, 8/8;
  • prompt == one-row decode at 32/32 cut points, and resumed == fresh 32/32;
  • restart, fixed pages == paged, and evictions from a 0.1 GiB cache all match;
  • after every drain the pool reads 520/520 free.
• Machine code: every existing function is unchanged on sm_80/86/89/120/121, under nvcc 13.0 and 13.2.
• Contract: the 4 new kernels pass, and their 4 controls go red. Limit records pass, 10/10 per toolkit.
• lean_check 0 new; zig build test passes.

Speed (GB10)

decode, 1 / 4 / 8 streams	112.8 / 420.0 / 767.4 tok/s
cold TTFT, 2k / 8k / 32k	0.150 / 0.615 / 2.84 s
warm turn, 10.25k tokens with 10.14k cached	0.049 s (cold 0.81 s)

Questions

1. Default served context and pool size: the model allows 262,144; the default here is 32,768. What would you like?
2. The per-head gate runs as its own launch after attention (headGate). Fusing it into the attention epilogue would save a pass over o. Would you rather have it now or later?
3. The NVFP4/FP8 schemes in the shared admission: is extending Blivion's weights.zig the right place, so Metal refuses these exports by name?

[ashhart]

Thanks @jschmied. A float64 reference that agrees with the official modelling code at every position is the evidence we want. Admitting only the export's exact ignore list keeps the scheme honest.

Before this leaves draft:

1. It stacks on an old copy of the admission. cb2d77003 still adds the root build-lfm3.zig, which lfm3: add native Metal vision support above the text decoder #1502 has since folded into build.zig. Please rebase onto the admission as it lands, and onto main once lfm2_moe: serve LFM2.5-24B-A2B on CUDA (plain serve, shared kernels; stacked on #1377) #1492 has.
2. zig/src/families/lfm3/cuda_backend.zig is lfm2_moe CUDA: attention pages from the shared KV pool, and the prompt cache (stacked on #1492) #1503's 426 lines with the names changed; cuda_snap.zig and cuda_native.zig are mostly the same as lfm2_moe CUDA: attention pages from the shared KV pool, and the prompt cache (stacked on #1492) #1503's too. Please share one lane backend, and the state walk, between the two Liquid families, generic over the model and the forward. liquid_conv.cuh already shares the conv that way.
3. pattn_kernel<128, 8, 4, 8> has a limit record (lfm3_cuda_records.zig:11) but no gate. Please gate it against contract_attn.zig's float64 reference, with a planted control, as we've asked on lfm2_moe: serve LFM2.5-24B-A2B on CUDA (plain serve, shared kernels; stacked on #1377) #1492 for head 64.
4. Each kept state allocates a device buffer and frees it behind a stream wait (cuda_snap.zig:61, :75-76), as in lfm2_moe CUDA: attention pages from the shared KV pool, and the prompt cache (stacked on #1492) #1503. The same fix applies.
5. Please commit the reference, and list each miss with its tokens and both margins.
6. Please label the mode weight-only: the export's static FP4 input scales are read but not applied (cuda_weights.zig:118).
7. Please state the weights' licence.

To your questions:

• Keep 32,768 as the default, as LFM2 does, and raise it once 64k and longer contexts are measured.
• Fuse the head gate later, in its own PR, with an A/B.
• Yes, one admission for both backends. Whichever of this and lfm3: add native Metal vision support above the text decoder #1502 lands second should make Metal refuse modelopt_nvfp4 by name, with a test.

Then we'll run the contract and its limits. We'll serve the NVFP4 export on a GB10: drafted against plain, concurrent against solo, prompts across the sliding window, and resumed against fresh.