[diwakergupta]

What this changes

A dense EXL3 checkpoint's bf16 embedding rows (2.37 GiB on the 27B) now load into mapped host memory, but only when the card has no room for them. In every configuration that fits today, the load and the default path are main's.

This is the host-mapped embedding from #1476, made automatic as @ashhart suggested when parking #1476. Unconditional mapping slowed decode on the GB10, so it's now a fallback rather than the default. Its value is on a 24 GB card: there it lets the 27B with DFlash2 serve a 98k or 128k-token window that main refuses to start.

• The rule (cuda_host_embed.zig): the rows go to host memory when a discrete card's room is less than the weights and scratch, one stream at the asked window, the DFlash2 drafter's weights and one stream of its ring, taps and frame scratch, plus 256 MiB for the load's other buffers (about 85 MB measured: kernel modules and the EXL3 kit). Unified memory (the GB10) never maps them: its host memory is the same memory.
• Where it is decided. qwen3_5's Options gain room (the server's room probe, which gemma4 already takes) and drafter. open passes them to Engine.initFit, which decides between the loader's measure pass and the arena allocation. Engine.init is initFit without a fit, so its other callers are unchanged.
• The drafter's stream before it loads. Draft.streamBytesFor(config, hidden) uses the drafter's own layout() for one frame, plus its ring and the target's taps. Without it, the rule missed the 98k case by the drafter's 0.35 GiB.
• Loading the rows. cuda_exl3_weights skips the rows in the arena when Engine.host_embed is set, and copies them into that pinned, device-mapped buffer. The engine frees it in deinit and on a failed load.
• The server logs one line when it maps: qwen3_5: embedding rows read from host memory: no room beside a 98304-token stream.

Receipt

• Environment. RTX 3090 Ti (24 GB, sm_86). turboderp/Qwen3.8-27B-exl3 branch 3.50bpw (rev 8351c54e), z-lab/Qwen3.8-27B-DFlash2 (rev 50307d4c) at 4 bits, bf16 KV cache. Both binaries came from zig build native -Dsm=86, one from main (a0d24896) and one from this branch.

• What starts, by --context. Each run was a fresh server, with the start-up line's numbers:

  --context	main	this PR
  32,768	model 15.66 GiB, 2 streams	the same: rows on the card
  65,536	model 15.66 GiB, 1 stream	the same: rows on the card
  98,304	refuses: a 6.90 GiB stream, 6.56 GiB left	rows mapped, model 13.29 GiB, 1 stream
  131,072	refuses: an 8.90 GiB stream, 6.56 GiB left	rows mapped, model 13.29 GiB, 1 stream
  163,840	refuses	rows mapped, still refuses: a 10.90 GiB stream, 8.93 GiB left

• Exactness with the rows mapped (--context 98304). tools/bench_concurrent.py --levels 1 --tokens 384 --reps 3 --alone --serial, the default code and chat cells (greedy and sampled), and --prompts with two long prompts (5,080 and 2,278 tokens):

  • Drafted == "draft": false and every repeat equal: 0 unequal, 0 failed.
  • token_sha256 equals main's (at --context 32768) in 72/72 replies.

• Decode speed with the rows mapped, against main on the card, per-stream tok/s (a median of 3). Here it costs 0–1.2%; main can't run at this window at all:

  Cell	main, 32k, rows on the card	this PR, 98k, rows mapped	change
  code, sampled	190.2	190.0	−0.1%
  chat, sampled	116.1	114.7	−1.2%
  code, greedy	183.7	183.7	0.0%
  chat, greedy	108.0	107.8	−0.2%
  long prompts, sampled	112.7	113.0	+0.3%
  long prompts, greedy	116.1	116.1	0.0%

• Default path. At 32k and 64k the rows stay on the card and the same model bytes load as on main. The code that runs per token is main's.

• Tests. The full checklist suite (zig build test test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon test-cuda-host) plus test-qwen35-formats-host: 481/481 steps succeeded. lean_new.py against main: 0 new problems. New host test: the rows stay on the card unless a discrete card's room is short of the device bytes and one stream (cuda_host_embed.zig, run by test-qwen35-formats-host).

• Not run. GB10: unified memory never maps, so its path is main's. Metal: not affected. MLX and MoE EXL3 checkpoints: the rule is EXL3-only and the MLX loader doesn't read it; a MoE EXL3 checkpoint takes the same rule but wasn't served. Other drafters: the rule sizes only a DFlash2 drafter and counts nothing for any other.

Checklist

• Every token still goes through the lane rounds.
• Drafted output equals "draft": false, and the mapped rows give main's tokens.
• No precision traded for speed: the same bf16 rows, read from host memory.
• No GPU kernel changed.
• Prompt processing unchanged in bits.
• Platforms: CUDA sm_86 only.
• The new test covers the rule (its function is new, so there is no before).
• Comments and docstrings are one line.
• No personal data, local paths or AI attribution lines.

[ashhart]

Thanks @diwakergupta, this is the shape we asked for. Two things before it lands:

1. Could you rebase onto current main? The Options line in cuda_native.zig now also has rope_yarn.
2. Is there a --context that main starts on a 24 GB card but this branch would map? The rule adds 256 MiB beyond the arena, the stream and the drafter. Near main's limit, it could map rows that main keeps on the card, which would cost up to 1.2% decode. A start-up line at the largest window main accepts would answer it.