diff --git a/docs/recipes/README.md b/docs/recipes/README.md index f3e82f73b4..7d717e65dc 100644 --- a/docs/recipes/README.md +++ b/docs/recipes/README.md @@ -7,6 +7,7 @@ the native engine, or the Python engine frozen at 0.6.6. | --- | --- | | Nemotron 3.5 Lightning | [MLX](nemotron-3.5.md), [Intel Arc, experimental](xpu-arc.md) | | Nemotron 3.5 Super VL | [Native CUDA on one or two GB10s, images](nemotron-3.5-super-vl.md) | +| Nemotron 3.5 Bolt | [Native CUDA on one GB10, libbolt, images and video](nemotron-3.5-bolt.md) | | Qwen3.8-27B | [MLX, quantization and CUDA](qwen3.8-27b.md) | | Qwen3.5-2B | [Native Metal preview, tied affine 4-bit](qwen3.5-2b-native.md) | | Qwen3.8 Flash Next | [MLX prefill and CUDA](qwen3.8-flash-next.md), [CUDA images](flash-next-vision.md) | diff --git a/docs/recipes/nemotron-3.5-bolt.md b/docs/recipes/nemotron-3.5-bolt.md new file mode 100644 index 0000000000..873673bdb6 --- /dev/null +++ b/docs/recipes/nemotron-3.5-bolt.md @@ -0,0 +1,99 @@ +# Nemotron 3.5 Bolt + +[NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB) is a +74B-parameter Mamba-2, attention and MoE model (9B active) packed to 24 GB: IQ2_XXS experts, NVFP4 Mamba projections, +Q8_0 attention and latent projections. Its `config.json` is a `nemotron_h_omni` wrapper around a `nemotron_h_puzzle` +text model (hidden 4096, 88 blocks, 240 to 496 routed experts a layer, top-9), so the native CUDA host serves it as its +own family, `nemotron_h_puzzle`, on one GB10 (sm_121). The family is `zig/src/families/nemotron/bolt_native.zig`, a lane +backend over `libbolt.so`; the HTTP server, chat template, round loop, keyed sampler and prompt cache are the engine's. + +## Build and run + +`libbolt.so` is built from its CUDA sources, which this repository does not carry (they are +[MiaAI-Lab's](https://github.com/MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark/tree/main/engine)): + +```bash +zig build -Dnvcc=/usr/local/cuda/bin/nvcc -Dbolt-src=PATH/engine -Doptimize=fast install native bolt +tensorfold-native serve NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB --parallel 8 --context 262144 +``` + +`native/lib/libbolt.so` is found beside the server (`TENSORFOLD_BOLT_LIB` names another); the backend checks its C ABI +version at load. Images need `--vision`; videos also `libtfvideo` (`zig build video`). The drafter, +[LiLiCorr](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr), is named by `BOLT_DRAFT` +(a folder) or `--no-drafts` serves one row a round. `BOLT_KV=fp8` stores the attention cache as e4m3 with a scale per +token and head (4,224 bytes a token against 8,192), and `BOLT_KV_PIN=1` maps every stream's whole window at load. + +## What it serves + +- Up to 16 streams (`--parallel`) on one copy of the weights; a shared round carries every stream's verify window in one + forward. The window is the model's 262,144 tokens a stream; attention rows are mapped as a stream grows. +- LiLiCorr drafts verified in the lane rounds, seven tokens a window, drawn with the stream's keyed sampler. +- Prompt cache over the Mamba state, attention rows and drafter context (`snap_*`, any row, no lookahead). +- Images through the RADIO tower (`--vision`): each prompt piece encodes the images it crosses into the rows buffer, so a + request holds as many images as fit the window (`--vision-max-images`, up to 64). Videos through the same tower: frames + sampled at 2 a second (1 above 720 rows, at most 128), two to a tubelet through `video_embedder`, each tubelet labelled + "Frame i sampled at t seconds and frame j ..." as the checkpoint's processor writes it (`--vision-max-videos`). +- Structured output and forced tool calls: `response_format` (`json_schema`, `json_object`), `guided_*` fields and + `tool_choice` `required` or a named function. The server compiles the grammar (`libtfgrammar`, `zig build grammar`) to + per-row allowed-token bits; the backend redraws the rows of a verify window among those tokens (a masked top-k on the + GPU, `bolt_mask_rows`), hands the grammar its held drafts, and draws the first token again under the first mask. +- Not served: logprobs, speculative trees, int8 or int4 caches, `--slide`. + +## Exactness + +A stream's rounding is the same alone, among other streams and in a verify window of any width. Three things made it so: + +1. **Attention chunks follow the stream's own length.** A verify window merges partial softmaxes over chunks of keys. The + chunk is `win_chunk(base)`: 128 keys, doubled until 32 chunks reach the end of the sequence and its window. Before, the + chunks followed the number of streams in the block (about 32 splits for one stream, 8 for four) and the window's width, + so a stream among others rounded differently from the same stream alone. +2. **Decode GEMMs sum K in one order.** Dense GEMMs for blocks of up to 32 rows split K across four warps (interleaved + partial sums, then added); blocks over 32 rows used a kernel that walks K in one accumulator. A stream's rounding changed + when other streams grew the block past 32 rows. Decode and verify blocks now always use the K-split kernel (64 tokens + a CTA); the prompt pass keeps the faster large-block kernel. +3. **The plain step is a window.** The one-token step (no drafts) runs as a one-row verify window and is kept at once, so + it shares the arithmetic of drafted decoding. + +Checks in the engine's harness (not in this repository), run with 32-token GEMM tiles: the perplexity of four passages +scored alone and in a block of 2, 3, 4, 8 and 16 streams (windows of 1, 5 and 8 rows, up to 128 rows a block) is +identical to five decimals; sixteen sequences decoded in one block equal their solo runs for 768 of 768 tokens, plain +steps and 4-row windows alike. The final 64-token tile changes only the tokens a CTA holds, not the order K is summed +in, and is covered by the `bench_concurrent.py` table below. Perplexity on 1,499 tokens is 5.286. A stream's logits are +bit-identical whatever its neighbours hold (leak test). + +The prompt pass uses different kernels from decode by design; `--alone` compares decode. + +## Receipts + +TensorFold `46a08c18` plus this change; DGX Spark GB10, Ubuntu 24.04.4, kernel 6.17.0-1026-nvidia, driver 580.159.03, +CUDA 13.3.73; checkpoint `a326822af111290f2af5ac59ef4b5040347dabc4`, drafter `4c6a5613b6d887194a3a5493e3a18a5b3930bb86`; +`--parallel 8 --context 262144 --prompt-cache-gib 8 --vision`, bf16 cache with every stream's window mapped at load +(`BOLT_KV_PIN=1`), greedy, 256 tokens, 3 repetitions. + +`tools/bench_concurrent.py --levels 1,4,8 --alone --serial`: + +| Prompt | Streams | Aggregate | Each stream | Max TTFT | Concurrent equals solo (`--alone`) | Drafted equals `"draft": false` (`--serial`) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| code | 1 | 115.1 tok/s | 115.1 | 0.07 s | 3 of 3 | 1 of 1 | +| code | 4 | 207.9 tok/s | 54.8 | 0.45 s | 12 of 12 | 1 of 1 | +| code | 8 | 252.6 tok/s | 35.2 | 1.15 s | 24 of 24 | 1 of 1 | +| chat | 1 | 60.7 tok/s | 60.7 | 0.05 s | 3 of 3 | 1 of 1 | +| chat | 4 | 139.1 tok/s | 35.6 | 0.42 s | 12 of 12 | 1 of 1 | +| chat | 8 | 174.7 tok/s | 23.6 | 1.05 s | 24 of 24 | 1 of 1 | + +Cold prefill with `tools/bench_cache.py --require-cold` (0 cached tokens, `qualified: true`), measured on the build +before the exactness changes, which touch decode blocks only (not re-run since): TTFT 1.44 s at 2,048 tokens, 5.60 s at +8,192, 24.66 s at 32,768, 55.60 s at 65,536. + +Cost of the K-split kernel in the widest rounds (a shared round of N streams x R rows, harness, ms; before the change / +after): 8 x 8 rows 135.5 / 130.4, 16 x 4 rows 133.2 / 141.7, 16 x 8 rows (128 rows) 176.9 / 220.3. Rounds of up to 32 rows +are unchanged. + +## Limits of this change + +`libbolt` is a CUDA runtime library, not fatbins under the kernel contract, and its sources are outside this tree +(4 files, about 4,800 lines, past the lean checker's file and comment rules); they are in the +[recipe repository](https://github.com/MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark/tree/main/engine), which is +private for now. It targets sm_121 only and was run on one GB10. Whether the kernels belong in `zig/kernels/cuda` under +the contract, stay a separate repository, or go through another route is the maintainers' call; the Zig side here is the +part that does not depend on that answer. diff --git a/zig/build/bolt.zig b/zig/build/bolt.zig new file mode 100644 index 0000000000..f56d62462c --- /dev/null +++ b/zig/build/bolt.zig @@ -0,0 +1,27 @@ +//! libbolt, Nemotron 3.5 Bolt's forward pass: an optional library nvcc builds beside the server from -Dbolt-src. + +const std = @import("std"); + +const sources = [_][]const u8{ "bolt_kernels.cu", "bolt_mma.cu", "bolt_host.cu", "bolt_draft.cu" }; + +/// `zig build bolt -Dnvcc=NVCC -Dbolt-src=DIR`: native/lib/libbolt.so, which a Bolt checkpoint opens. +pub fn steps(b: *std.Build, nvcc: ?[]const u8) void { + const step = b.step("bolt", "libbolt (Nemotron 3.5 Bolt's engine, sm_121) from -Dbolt-src with -Dnvcc, beside the server"); + const src = b.option([]const u8, "bolt-src", "the directory of libbolt's .cu sources (bolt_host.cu and its siblings)"); + const tool = nvcc orelse { + step.dependOn(&b.addFail("zig build bolt needs -Dnvcc=PATH to nvcc").step); + return; + }; + const dir = src orelse { + step.dependOn(&b.addFail("zig build bolt needs -Dbolt-src=DIR, the folder holding bolt_host.cu").step); + return; + }; + const root: std.Build.LazyPath = .{ .cwd_relative = dir }; + const run = b.addSystemCommand(&.{ tool, "-shared", "-Xcompiler", "-fPIC", "-O3", "-gencode", "arch=compute_121,code=sm_121", "-I" }); + run.addDirectoryArg(root); + run.addArg("-o"); + const lib = run.addOutputFileArg("libbolt.so"); + for (sources) |name| run.addFileArg(root.path(b, name)); + run.addArg("-lcuda"); + step.dependOn(&b.addInstallFile(lib, "native/lib/libbolt.so").step); +} diff --git a/zig/build/cuda.zig b/zig/build/cuda.zig index 684d6c3cc9..26e993ec9d 100644 --- a/zig/build/cuda.zig +++ b/zig/build/cuda.zig @@ -3,6 +3,7 @@ const std = @import("std"); const exlr8_build = @import("exlr8.zig"); const golden_build = @import("golden.zig"); +const bolt_build = @import("bolt.zig"); /// Each .cu in zig/kernels/cuda (`src`, else `name`) with the flags its Python extension passes in `extra_cuda_cflags`. const Kernel = struct { name: []const u8, flags: []const []const u8, src: ?[]const u8 = null, arch_specific: bool = false }; @@ -111,6 +112,7 @@ pub fn targets(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.bu const prebuilt = b.option([]const u8, "fatbins", "absolute directory of prebuilt .fatbin files to embed"); const sms = b.option([]const u8, "sm", "SASS targets, comma separated (default 121; 80, 86, 89, 120 and 121 build)") orelse "121"; const strip = b.option(bool, "strip", "No debug info in the CUDA executables: no build machine paths leave with them") orelse false; + bolt_build.steps(b, nvcc); // the compiler's version text is an input of every fatbin, so a new nvcc rebuilds them all const version: ?std.Build.LazyPath = if (prebuilt == null and nvcc != null) blk: { const run = b.addSystemCommand(&.{ nvcc.?, "--version" }); diff --git a/zig/src/cli/hub.zig b/zig/src/cli/hub.zig index 99d6835a49..cccb1f1123 100644 --- a/zig/src/cli/hub.zig +++ b/zig/src/cli/hub.zig @@ -11,6 +11,7 @@ pub const Family = struct { model_type: []const u8, title: []const u8, formats: const titles = [_]struct { model_type: []const u8, title: []const u8 }{ .{ .model_type = "nemotron_h", .title = "Nemotron 3.5 Lightning" }, .{ .model_type = "nemotron_h_omni", .title = "Nemotron 3.5 Super VL" }, + .{ .model_type = "nemotron_h_puzzle", .title = "Nemotron 3.5 Bolt" }, .{ .model_type = "qwen4_exp", .title = "Qwen3.8 Flash Next" }, .{ .model_type = "glm5_next", .title = "GLM-5.3-Flash" }, .{ .model_type = "qwen3_5", .title = "Qwen3.5-2B or Qwen3.8-27B" }, @@ -114,14 +115,18 @@ pub fn configInt(a: Allocator, io: std.Io, dir: []const u8, key: []const u8) ?u6 return if (v == .integer and v.integer >= 0) @intCast(v.integer) else null; } -/// config.json's model_type, "unknown" when unreadable. +/// config.json's model_type, "unknown" when unreadable; an omni wrapper of a puzzle model (Bolt) is nemotron_h_puzzle. pub fn modelType(a: Allocator, io: std.Io, dir: []const u8) []const u8 { const text = readSmall(a, io, pathJoin(a, dir, "config.json") catch return "unknown") orelse return "unknown"; var parsed = std.json.parseFromSlice(std.json.Value, a, text, .{}) catch return "unknown"; defer parsed.deinit(); if (parsed.value != .object) return "unknown"; const t = parsed.value.object.get("model_type") orelse return "unknown"; - return if (t == .string) t.string else "unknown"; + if (t != .string) return "unknown"; + if (std.mem.eql(u8, t.string, "nemotron_h_omni")) if (parsed.value.object.get("llm_config")) |llm| if (llm == .object) { + if (llm.object.get("model_type")) |lt| if (lt == .string and std.mem.eql(u8, lt.string, "nemotron_h_puzzle")) return "nemotron_h_puzzle"; + }; + return a.dupe(u8, t.string) catch "unknown"; } /// config.json's quantization: bits and group size, when the checkpoint is MLX-affine quantized. @@ -196,6 +201,11 @@ test "cache layout helpers read a synthetic cache" { const snapshot = (try cachedSnapshot(a, io, hub, "Org/Name")).?; try std.testing.expect(std.mem.endsWith(u8, snapshot, "0123456789abcdef0123456789abcdef01234567")); try std.testing.expectEqualStrings("qwen4_exp", modelType(a, io, snapshot)); + try w.writeFile(io, .{ .sub_path = try std.fs.path.join(a, &.{ snapshot, "config.json" }), .data = "{\"model_type\": \"nemotron_h_omni\", \"llm_config\": {\"model_type\": \"nemotron_h_puzzle\"}}" }); + try std.testing.expectEqualStrings("nemotron_h_puzzle", modelType(a, io, snapshot)); + try w.writeFile(io, .{ .sub_path = try std.fs.path.join(a, &.{ snapshot, "config.json" }), .data = "{\"model_type\": \"nemotron_h_omni\", \"llm_config\": {\"model_type\": \"nemotron_h\"}}" }); + try std.testing.expectEqualStrings("nemotron_h_omni", modelType(a, io, snapshot)); + try w.writeFile(io, .{ .sub_path = try std.fs.path.join(a, &.{ snapshot, "config.json" }), .data = "{\"model_type\": \"qwen4_exp\", \"quantization\": {\"bits\": 6, \"group_size\": 32}, \"max_position_embeddings\": 65536}" }); const q = quantization(a, io, snapshot).?; try std.testing.expectEqual(@as(u64, 6), q.bits); try std.testing.expectEqual(@as(u64, 32), q.group); diff --git a/zig/src/core/engine_api.zig b/zig/src/core/engine_api.zig index 243b3386a4..3e2c13d957 100644 --- a/zig/src/core/engine_api.zig +++ b/zig/src/core/engine_api.zig @@ -2,6 +2,7 @@ const std = @import("std"); const lanes = @import("lanes"); const Allocator = std.mem.Allocator; +const video_offer = @import("video_offer.zig"); /// The server's name for one request, unique while the process lives. pub const tfvideo = @import("tfvideo.zig"); @@ -132,43 +133,10 @@ pub const Vision = struct { explain: ?*const fn (err: anyerror) ?[]const u8 = null, }; -/// A video's facts as the server's decoder reads them, and its frames on request. -pub const VideoSource = struct { - frames: u64, // the stream's frame count - rate_num: u64, // its average frame rate, num / den - rate_den: u64, - width: u32, - height: u32, - ctx: *anyopaque, - /// The frames at ``indices`` (ascending, distinct), each to ``sink`` in order. - decode: *const fn (ctx: *anyopaque, indices: []const u32, sink: FrameSink) anyerror!void, -}; - -/// Where a decoded frame goes: RGB rows of ``width`` * 3 bytes, ``stride`` bytes apart. -pub const FrameSink = struct { - ctx: *anyopaque, - take: *const fn (ctx: *anyopaque, k: usize, rgb: []const u8, width: u32, height: u32, stride: usize) anyerror!void, -}; - -/// A video prepared for the tower: each group of frames an image, and each group's time in seconds. -pub const PreparedVideo = struct { groups: []PreparedImage, times: []f64 }; - -/// Video input an engine serves (``Vision.video``): how a decoded video becomes images and how the prompt holds them. -pub const VideoOffer = struct { - /// ``src`` at most ``max_tokens`` tokens and ``max_frames`` sampled frames; allocated in ``a``. - prepare: *const fn (ctx: *anyopaque, a: Allocator, src: VideoSource, max_tokens: u32, max_frames: u32) anyerror!PreparedVideo, - /// The token the chat template writes once per video, expanded to its groups. - video_token: u32, - /// The tokens around each group's image tokens; the group's time follows as text. - group_open: u32, - group_close: u32, - /// The tokens the template wraps a video in (the video token among them): text that spells one stays text. - markers: []const u32 = &.{}, - /// Video tokens one request may hold in all, videos, and frames sampled a video. - video_tokens: u32, - max_videos: u32, - max_frames: u32, -}; +pub const VideoSource = video_offer.VideoSource; +pub const FrameSink = video_offer.FrameSink; +pub const PreparedVideo = video_offer.PreparedVideo; +pub const VideoOffer = video_offer.VideoOffer; /// --vision: image and video input's limits (``Open.vision``); image_tokens_set: the flag given, over a family default. pub const VisionOpen = struct { image_tokens: u32 = 4096, image_tokens_set: bool = false, max_images: u32 = 4, video_tokens: u32 = 16384, max_videos: u32 = 2, max_frames: u32 = 256 }; diff --git a/zig/src/core/video_offer.zig b/zig/src/core/video_offer.zig new file mode 100644 index 0000000000..1a65e0e607 --- /dev/null +++ b/zig/src/core/video_offer.zig @@ -0,0 +1,45 @@ +//! A video as an engine reads it: the decoder's facts and frames, the groups it prepares, and what the prompt holds. +const std = @import("std"); +const api = @import("engine_api.zig"); +const Allocator = std.mem.Allocator; +const PreparedImage = api.PreparedImage; + +/// A video's facts as the server's decoder reads them, and its frames on request. +pub const VideoSource = struct { + frames: u64, // the stream's frame count + rate_num: u64, // its average frame rate, num / den + rate_den: u64, + width: u32, + height: u32, + ctx: *anyopaque, + /// The frames at ``indices`` (ascending, distinct), each to ``sink`` in order. + decode: *const fn (ctx: *anyopaque, indices: []const u32, sink: FrameSink) anyerror!void, +}; + +/// Where a decoded frame goes: RGB rows of ``width`` * 3 bytes, ``stride`` bytes apart. +pub const FrameSink = struct { + ctx: *anyopaque, + take: *const fn (ctx: *anyopaque, k: usize, rgb: []const u8, width: u32, height: u32, stride: usize) anyerror!void, +}; + +/// A video for the tower: each group of frames an image, its time in seconds, and the words before it (``labels``). +pub const PreparedVideo = struct { groups: []PreparedImage, times: []f64, labels: []const []const u8 = &.{} }; + +/// Video input an engine serves (``Vision.video``): how a decoded video becomes images and how the prompt holds them. +pub const VideoOffer = struct { + /// ``src`` at most ``max_tokens`` tokens and ``max_frames`` sampled frames; allocated in ``a``. + prepare: *const fn (ctx: *anyopaque, a: Allocator, src: VideoSource, max_tokens: u32, max_frames: u32) anyerror!PreparedVideo, + /// The token the chat template writes once per video, expanded to its groups. + video_token: u32, + /// The text the template writes for a video, swapped for ``video_token``'s text before encoding (Omni's `