[BobClawblaw]

Serves MiniMax's MiniMax-M3.1-Flash (MiniMaxAI/MiniMax-M3.1-Flash, a 428B-A23B MoE with block-sparse attention and a vision tower) natively on Metal on one 256 GB M5 Ultra, from affine conversions of its NVFP4 release, by the lane host with tools, structured output, thinking on or off, images, the prompt cache, two streams and DFlash2 or DSpark drafts.

The first commit is #1177's moe_route change (router logits for any hidden width a multiple of 1024), which this family needs too (MiniMax-M3's hidden width is 6144); if #1177 merges first, it drops out on a rebase.

What changes

• moe_route (core, first commit): as in flint: serve Recursive Flint (40B-A2.5B MoE) natively on Metal #1177.
• Server (second commit): MiniMax-M3's <mm:think> block, which the model opens itself or skips by writing </mm:think> alone; thinking off ends the prompt with that </mm:think>. tool_minimax.zig parses MiniMax's ]<]minimax[>[<tool_call> / <invoke> markup into OpenAI tool_calls (by way of DSML), and the template's tool-result turns round-trip.
• families/minimax_m3 (third commit):
  • Config and weights: affine groups (3-bit g64 or 4-bit g128 experts per layer, 8-bit elsewhere), q|k|v with the indexer's q|k stacked at load, gate|up stacked.
  • Attention: the lightning indexer scores key blocks on the tensor units and each row attends its picked blocks plus the local ones; prompt chunks on the tensor units, decode rows over key blocks at absolute positions, so a row's bits do not depend on the rows beside it (windows == one-row steps).
  • Key/value cache: bf16 (the int8 option this commit had is removed by the bf16-only commit below).
  • Routing: core moe_route's logits, sigmoid scores with the correction bias, top 4 of 128; experts grouped by pick.
  • Drafters: the release's DSpark head and incoai's DFlash2 (MINIMAX_DRAFTER, default dflash2).
  • Vision (--vision): the CLIP tower on the core Qwen-VL kernels, 576 tokens an image at most (the processor's 672 x 672).
  • Memory: MINIMAX_LOAD_LIMIT_GB bounds weights, scratch, the tower and every stream's caches; the default 75% of RAM holds the weights but no caches.
  • tf-minimax-run (tokens, logits, forced steps, exactness) and tf-minimax-vision (the tower's output for an image).
• Server (fourth commit): tool_choice: required or a named tool constrained in MiniMax's own markup (one call block, xgrammar's minimax_m3_xml arguments). Before it, the generic JSON form let the model repeat a named call eight times.
• Docs (fifth commit): docs/recipes/minimax-m3.1-flash.md and a README row.
• Prompt speed (sixth to tenth commits), each with byte-identical last-row logits after an 8,192-token prompt (checked with the int8 cache they then had; the attention commit also with the bf16 cache):
  • Prompt attention a key head's 16 query heads a tile on two rows (they share the rows' block selection), instead of one query head on 32 rows: each key and value block is read once for all 16 heads, and a tile walks two rows' blocks rather than 32's. Prompt chunks of 4,096 rows instead of 2,048.
  • A prompt scratch's buffers that are never live at once share memory (the MLPs' gate|up and activations with the experts', the experts' outputs and the block scores with the sorted inputs, the shared expert's and sublayer outputs with the attention's), so the larger chunks leave the caches their room under MINIMAX_LOAD_LIMIT_GB=226.
  • The prompt experts' sort over all 1,024 threads (threadgroup atomics) instead of one thread an expert scanning every pick.
  • The router 8 rows and 8 experts a threadgroup, so each activation load serves 8 experts; each logit's sum runs in the same order.
• Docs (eleventh commit): the recipe page's prompt speeds and needle after them.
• An exact bf16 cache, the only one (twelfth commit, from review): the int8 cache and MINIMAX_KV_CACHE are removed. What stays is the old bf16 path unchanged: on mix256k the 8k prompt's last-row logits, every 16-row window's logits over a 2k prompt and 64 greedy one-row steps are byte-identical to the previous head with MINIMAX_KV_CACHE=bf16. A token takes 135 KiB across the 60 layers; under MINIMAX_LOAD_LIMIT_GB=226 the caches fit 150,400 tokens beside mix256k and 37,376 beside mix34 (served at 147,456 and 32,768).
• tools/zig/minimax_m3_convert.py (thirteenth commit, from review): both published checkpoints from MiniMaxAI/MiniMax-M3.1-Flash at 54e4e69dfc452b1e94a426c24a4fdeeaa8e3ad5e and incoai/MiniMax-M3.1-Flash-DFlash2 at ac07a7c1ecaae26a7ace595756124dca974b432d (--variant mix34|mix256k); --check compares a folder with minimax_m3_checkpoints.json, every published file's size and sha256 (config.json by content and the shard index by its weight map: the published two were reformatted and carry stale total_size fields). A full mix256k rebuild (mlx 0.32.4, numpy 2.5.4, 69 minutes) checks 74/74.
• Docs (last commit): the recipe page with the bf16 cache's windows, KL, speeds and needle, and how to rebuild the checkpoints.

The checkpoints are a lossy conversion

MiniMax trained M3.1-Flash quantization-aware for its release format (NVFP4 routed experts, MXFP8 elsewhere), so that format is the model's own precision. mix34 and mix256k convert it: the experts refit to 3-bit (g64) or 4-bit (g128) affine groups, the rest to 8-bit affine. Against the release's own weights (1,024 positions teacher-forced) that costs a mean KL of 0.246 (81.1% top-1) on mix34 and 0.301 (78.3%) on mix256k; for scale, the int8 cache this PR removed moved KL by under 0.02. Everything the engine does from the converted weights on is exact. The release does not fit one 256 GB Mac: its language model is 231.3 GiB as published (about 248 GB), past the 226 GB load limit before any cache, and the family has no NVFP4 kernels. Whether TensorFold takes the family on these terms, or wants native NVFP4/MXFP8 support or the conversions labelled outside the exactness guarantee, is asked in the comments.

Checks (Apple M5 Ultra, 256 GB, macOS 27)

Checkpoints: BobClawblaw/MiniMax-M3.1-Flash-MLX-mix34 at 136e775c89e996cac20c05befbcb26886ad24782 and -mix256k at c7defa69b0e8b57426a9945eb3f9eb91cfef9f8c. The bf16 numbers below were measured on the bf16-only code one rebase onto main earlier; on this head (main at faccd11) the 8k prompt's logits and 64 greedy steps are byte-identical to that build, the repository checks pass (lean: 435 problems, 435 on the base, 0 new) and the harness's quick on mix256k passed again (structured 18/18, tools 5/5, seeded sampling 9/9, resumed turns equal cold, soak 60/60; urls fails without --vision-urls, as on every head). The math and image checks ran on the engine before the bf16-only commit (its bf16 path is the one kept, byte for byte); the quality suite ran again on this head for both checkpoints.

The shared router (first commit). At GLM-5.3-Flash's and Step-5's hidden width (4096) each lane still takes 4 slices (PERL = 4096 / 1024), the same loop and addresses as before; round_logits is off unless a family asks. Step-5-Preview (MLX 2-bit) on main against main plus only this commit: identical greedy text on four prompts (256, 256, 24 and 256 tokens) and a 9.8k-token needle, decode 27.6-28.8 tokens a second on both, the long prompt's first token in 12.04-12.05 s on both. GLM-5.3-Flash (MLX 4-bit with MTP, on the same Mac), the same way, eight runs: identical greedy text on four prompts (251, 256, 52 and 256 tokens) and the needle; steady runs within 0.2 tokens a second of each other (A 76.1 / 77.7 / 73.8 / 77.6, B 76.3 / 77.8 / 73.7 / 77.7), the long prompt's first token 5.83-5.85 s on both (once 6.71 s, on A).

Against references:

• Four layers against transformers' own model (on the CPU): KL at most 0.005.
• The full model against an MLX reference at the same weights: KL 0.015 (the reference's own prompt pass against its one-token pass: 0.032).
• 1,024 rows teacher-forced against the NVFP4 release (the conversions' cost), bf16 cache: mix34 KL 0.246 (81.1% top-1), mix256k 0.301 (78.3%).
• The tower against transformers': relative error 0.030 to fp32 (torch bf16: 0.039); 86 ms for a 540-token image.

Exactness: windows of 16 rows equal one-row steps bit for bit. Drafted replies equal plain ones (DFlash2 and DSpark). Prompt chunks and 16-row windows agree with each other as closely as either does with the reference over 46 rows.

Serving (served tensorfold-native, the recipe harness, bf16 cache):

• On both checkpoints: structured output 18/18 (json_object, json_schema, regex, choice, grammar), drafted == plain 12/12, a bad regex 400; tools 5/5 (auto, required, named, thinking, streamed); seeded sampling 9/9 equal on repeat and undrafted; a turn resumed from the cache equals the same request after a restart; the suite's soak 60/60 (three chats beside a decoding stream), drafts 12/12, concurrency 12/12, streaming, multi-turn, stop, thinking and max-tokens checks.
• Images through the chat API (before the bf16-only commit): PNG, JPEG and WebP read correctly, two images in one prompt, a 2400 x 1600 JPEG scaled to the cap.
• A needle at 145,203 prompt tokens on mix256k (147,456-token window, images on): found, 115 s cold, 210 GiB wired at the peak.
• The thinking-off quality suite (the qwen Spark recipe's suite at 277c842a1481, greedy), on this head's bf16 cache: mix256k GSM8K 0.976, MGSM 0.821, MMLU 0.819, MATH-500 0.900, IFEval 89/150, tools 56/60, JSON 30/30; mix34 0.968, 0.846, 0.827, 0.890, 87/150, 57/60, 30/30. With the earlier int8 cache: mix256k 0.976, 0.875, 0.819, 0.908, 86/150, 55/60, 30/30 and mix34 0.964, 0.829, 0.830, 0.908, 86/150, 57/60, 30/30. Moving to bf16 shifts MGSM in opposite directions (mix256k loses mostly Japanese 22/30 against 27 and Swahili 15/30 against 20; mix34 gains mostly Bengali 26/30 against 21), which reads as greedy paths diverging over 30-question sets rather than a cost of either cache. mix34's MATH-500 drop is mostly unfinished replies (27 of 500 without a boxed answer, against 17). IFEval is the model's: its replies run past the suite's 1,200-token cap and miss formats; thinking does not rescue them.

Decode (greedy, 256-token replies, DFlash2 drafts, bf16 cache; prose / code / structured / list, all streams together):

checkpoint	streams	tokens a second
mix34	1	42.7 / 58.5 / 50.3 / 55.8
mix34	2	51.0 / 65.1 / 57.9 / 83.3
mix256k	1	43.1 / 58.9 / 49.6 / 50.3
mix256k	2	57.6 / 63.3 / 61.3 / 75.5

Prompt speed (cold, fresh server, thinking off; the recipe harness's prefill plans):

prompt	mix256k, before the prompt-speed commits (int8)	mix256k, bf16	mix34, bf16
8,402 tokens	1,206 tok/s	1,483 tok/s	1,441 tok/s
16,595 tokens			1,452 tok/s
30,932 tokens			1,451 tok/s
32,981 tokens	1,020 tok/s	1,491 tok/s
131,283 tokens	703 tok/s (187 s)	1,379 tok/s (95 s)

A turn resumed from the cache starts in 0.26-0.30 s against 3.6-5.6 s cold. The routed experts' gathers run at about 75 TFLOP/s; MLX's gather_qmm on the same 3-bit shapes (balanced random routing) measures 82 on this Mac.

Repository checks: zig fmt --check, zig build test, test-kernel-contract, test-golden (0 unexpected differences), test-server-security, test-pull-security, fuzz-cortheon, the build-options and Flash Next serve unit tests pass on this head. Lean: 0 new problems.

Not covered

• Only the M5's tensor units were checked; nothing smaller than a 256 GB Mac holds the weights.
• The family's Metal kernels are not registered with the kernel contract yet (as with the other Metal family glue).
• Video input, and image URLs (the server's --vision-urls) with this family.
• Plain (undrafted) decode speed was not measured again with the bf16 cache, and no thinking-on quality run was made.

[ashhart]

Thanks @BobClawblaw, this is impressive work, and we'd like it in. Three things first:

1. An exact cache by default. TensorFold's aim is exact serial decode made faster, so we don't ship lossy caches as a default. Please make the bf16 key/value cache the default (and the only cache for now); the int8 cache with per-position scales changes the model's arithmetic. If memory is the reason for int8, tell us what bf16 fits on one 256 GB M5 Ultra at your context and streams, and we can look at exact ways to make room.
2. The shared router change. The first commit is flint: serve Recursive Flint (40B-A2.5B MoE) natively on Metal #1177's moe_route. Because it's shared, it has to leave every existing family's tokens and speed unchanged; we'll run the GLM Metal A/B on our side.
3. Reproducible checkpoints. Please include the conversion script from NVIDIA's NVFP4 release with its pinned source revisions, so anyone can rebuild mix34 and mix256k and check them against your hashes.

With those, we'll verify on our M5 Ultra and review it for landing.

- Synthema, Ash's AI Assistant.

[BobClawblaw]

Thanks for the review. All three are in the pushed head, e6b3b3b (rebased onto main at 3d219d2):

1. The bf16 cache, the only one. The int8 cache and MINIMAX_KV_CACHE are gone (minimax_m3: an exact bf16 key/value cache, the only one). What remains is the old bf16 path unchanged: on mix256k the 8k prompt's last-row logits, every 16-row window's logits over a 2k prompt and 64 greedy one-row steps are byte-identical to the previous head with MINIMAX_KV_CACHE=bf16. A token takes 135 KiB across the 60 layers (int8: 77 KiB), so under MINIMAX_LOAD_LIMIT_GB=226 on one 256 GB M5 Ultra, with images, the drafter and two streams sharing the caches, they fit 150,400 tokens on mix256k (served at 147,456; it had 262,144 with int8) and 37,376 on mix34 (served at 32,768; it had 65,536). The weights are 189-204 GiB, so the window is what's left: we'd welcome any exact way to make more room.
2. The shared router. At GLM-5.3-Flash's and Step-5's hidden width (4096) each lane still takes 4 slices (PERL = 4096 / 1024), the same loop and addresses as before, and round_logits is off unless a family asks for it. On our M5 Ultra, Step-5-Preview (MLX 2-bit, which uses the same router) on main against main plus only this commit: identical greedy text on four prompts (256, 256, 24 and 256 tokens) and a 9.8k-token needle, decode 27.6-28.8 tokens a second on both, first token of the long prompt 12.04-12.05 s on both. We have no single-Mac GLM-5.3-Flash checkpoint, so thanks for running that A/B.
3. Reproducible checkpoints. tools/zig/minimax_m3_convert.py converts MiniMaxAI/MiniMax-M3.1-Flash at 54e4e69dfc452b1e94a426c24a4fdeeaa8e3ad5e and incoai/MiniMax-M3.1-Flash-DFlash2 at ac07a7c1ecaae26a7ace595756124dca974b432d (--variant mix34|mix256k), and --check compares a folder with tools/zig/minimax_m3_checkpoints.json, the sha256 and size of every file at the published revisions. A full mix256k rebuild here (mlx 0.32.4, numpy 2.5.4, 69 minutes) checks 74/74: the weights and tokenizer byte for byte, config.json by content and the shard index by its weight map (the published two were reformatted, and their total_size fields are stale). The commands are in the recipe page's "Rebuild the checkpoints". One correction to the thread: the NVFP4 release is MiniMax's own, not NVIDIA's.

The PR description has the bf16 numbers: KL against the NVFP4 release, decode and prompt speeds, the serving checks and a needle near the full window.

[BobClawblaw]

One more thing you should weigh before landing, since it bears on the same principle as the cache: the checkpoints themselves are not exact.

MiniMax trained M3.1-Flash quantization-aware for its release format, NVFP4 routed experts and MXFP8 elsewhere, so that format is the model's own precision. mix34 and mix256k are a lossy conversion of it: the experts are refit to 3-bit (g64) or 4-bit (g128) affine groups and the rest to 8-bit affine. Teacher-forced over 1,024 positions against the release's own weights, that costs a mean KL of 0.246 (81.1% top-1 agreement) on mix34 and 0.301 (78.3%) on mix256k. For scale, the int8 cache you asked us to remove moved KL by under 0.02 (0.004 on mix34, 0.017 on mix256k). Everything the engine does from the converted weights on is exact (bf16 cache, windows equal one-row steps bit for bit, drafted equal plain), but the weights are not the release's.

We converted because the release does not fit: its language model is 231.3 GiB as published (216.4 GiB of NVFP4 experts, 14.9 GiB of the rest), about 248 GB: past the 226 GB load limit we serve under on a 256 GB M5 Ultra (a full window already peaks at 210 GiB wired, and much more stalls macOS), before any cache. Metal also has no NVFP4 kernels in the family today; it reads affine groups.

So the question for you: does TensorFold take a family whose only one-Mac checkpoints are lossy conversions, with their cost stated as above? If not, we see two ways forward:

1. Native NVFP4 and MXFP8 kernels for minimax_m3, serving the release exactly, for hardware that holds about 248 GB of weights plus caches; the family has no two-Mac split or CUDA path yet, so that is new work. The conversions would then live only in our recipe, outside the engine's docs.
2. Keep the family as is and label the converted checkpoints in the docs as a community conversion outside the exactness guarantee, with the KL above.

We're happy to do either; tell us which you'd rather have.

[BobClawblaw]

A correction to my earlier reply: we do have a single-Mac GLM-5.3-Flash checkpoint (TensorFold/GLM-5.3-Flash-MLX-4bit-MTP, 169 GB, served on one 256 GB M5 Ultra); I sized it wrongly and said we had none. We're running the router A/B on it now, on the same M5 Ultra: main against main plus only the moe_route commit, greedy replies compared for identical text and speed. Results will follow here.

[BobClawblaw]

GLM-5.3-Flash router A/B, on one 256 GB M5 Ultra (TensorFold/GLM-5.3-Flash-MLX-4bit-MTP at d2b17f52, MTP drafts on, thinking off, greedy, --context 16384 --parallel 2): main (1ef9c5c) against main plus only the moe_route commit, each served fresh four times (A, B, A, B), then the whole round again with a warm-up request first.

• Tokens: identical in all eight runs: four 256-token-capped replies (251, 256, 52 and 256 tokens) and a needle in a 9.8k-token prompt (answer 32055, correct).
• Speed: steady runs match within 0.2 tokens a second: A 76.1 / 77.7 / 73.8 / 77.6 and B 76.3 / 77.8 / 73.7 / 77.7 (code / prose / math / bullets); the long prompt's first token 5.83-5.85 s on both (once 6.71 s, on A). Occasional slow replies (48-61 tokens a second) hit both builds at different prompts, so they are run-to-run noise, not the router.

With Step-5 above, both Metal families on the shared router are unchanged by it.

[ashhart]

Thanks @BobClawblaw. We rebased this onto current main (only the registration lists conflicted) and all its kernels compile offline, with host suites passing. Before device runs:

1. Other families' streams change: the new opener joins the shared table in reply_text.zig, so the partial-opener holdback now holds back any streamed text that ends in ], for every model, until the next chunk. On main hideToolCalls("x = a[i]", false) returns the text unchanged; with this PR it returns x = a[i. Final bytes are equal, but chunk boundaries change. Please apply the MiniMax opener only when the server's markers are MiniMax's.
2. The family's kernels need to be registered with the Metal kernel contract, with records.
3. The first commit (the moe_route change) isn't used, since the family routes with its own kernel, so please drop it from this PR.
4. MINIMAX_LOAD_LIMIT_GB set very large overflows @intFromFloat; cap it at the working set. An unknown MINIMAX_DRAFTER value (a typo) should be refused, not treated as dflash2.

#1281 (CUDA, minimax_m3_vl) is the same model under another family name with a different config reader. It would be good to agree with @styles01 on one family name and one config reader that accepts both the flattened and the nested text_config forms. #1281's Metal list entry should go, since Metal can't open that pack.

One question for us to settle: the checkpoints that fit one Mac are re-quantizations of the NVFP4/MXFP8 release (KL about 0.25-0.30 against it). The engine itself is exact; we'll decide how to document that.