[diwakergupta]

What this changes

The qwen3_5 engine's scratch rows for the two mixers and the dense MLP now share one region of the arena. A layer runs its mixer, then its MLP, and only h, x, xs and r pass between them, so the delta mixer's rows, the attention mixer's rows and the dense MLP's gate, up and act can start at the same offset. The region takes the largest of the three.

• cuda_scratch.zig carves the scratch: Take gains mark, rewind and past to lay the three regions over one another. The attention mixer's y and mix are the delta mixer's rows. Engine.scratch is now this file's function.
• MoE checkpoints keep their routed scratch after the shared region, as before.
• On the 27B this frees 0.46 GiB of device memory (model 15.66 → 15.20 GiB with DFlash2).

This is #1476 cut down, as @ashhart suggested: the host-mapped embedding is dropped from this PR. It slowed decode on the GB10, and it now loads only when the card has no room for it, in its own PR.

Receipt

• Environment. RTX 3090 Ti (24 GB, sm_86). turboderp/Qwen3.8-27B-exl3 branch 3.50bpw (rev 8351c54e), z-lab/Qwen3.8-27B-DFlash2 (rev 50307d4c) at 4 bits, bf16 KV cache, --context 32768. Both binaries came from zig build native -Dsm=86, one from main (a0d24896) and one from this branch. Each ran twice, alternating (main, this, main, this), on a restarted server.

• Exactness. tools/bench_concurrent.py --tokens 384 --reps 3 --alone --serial, the default code and chat cells (greedy and sampled) at --levels 1,4, and --levels 1 with --prompts of two long prompts (5,080 and 2,278 tokens):

  • Concurrent == alone and drafted == "draft": false in every cell on both binaries: 0 unequal, 0 failed.
  • token_sha256 of this branch equals main's in 528/528 replies.

• Decode speed. Per-stream tok/s, runs 1 / 2 (each a median of 3), with tokens per round:

  Cell	main	this PR	change	tokens/round main	tokens/round this PR
  code, sampled, 1 stream	190.2 / 189.6	190.0 / 190.3	+0.1%	6.86 / 6.86	6.86 / 6.86
  code, sampled, 4 streams	131.1 / 132.4	130.5 / 130.7	−0.9%	5.26 / 5.30	5.26 / 5.26
  chat, sampled, 1 stream	116.1 / 115.7	116.0 / 116.1	+0.1%	4.22 / 4.22	4.22 / 4.22
  chat, sampled, 4 streams	89.1 / 89.3	89.0 / 88.0	−0.8%	3.57 / 3.64	3.64 / 3.59
  code, greedy, 1 stream	183.7 / 182.6	183.7 / 183.0	+0.1%	6.62 / 6.62	6.62 / 6.62
  code, greedy, 4 streams	129.1 / 128.6	129.1 / 128.6	0.0%	5.16 / 5.16	5.16 / 5.16
  chat, greedy, 1 stream	108.0 / 107.5	107.9 / 107.4	−0.1%	3.92 / 3.92	3.92 / 3.92
  chat, greedy, 4 streams	88.3 / 88.1	88.2 / 88.0	−0.1%	3.56 / 3.56	3.56 / 3.56
  long prompts, sampled	112.7 / 112.7	112.7 / 113.0	+0.1%	4.57 / 4.57	4.57 / 4.57
  long prompts, greedy	116.1 / 117.3	116.3 / 113.1	−1.7%	4.68 / 4.74	4.68 / 4.57

  • Where tokens per round are equal, every cell is within ±0.1%.
  • The cells that differ are the ones whose tokens per round moved between runs. The sampled 4-stream cells move on main too (5.26 / 5.30), and in long greedy run 2 this PR's round count (4.57) is below both of main's.

• Prompt speed. Three distinct cold ~8,145-token prompts per run (cached_tokens 0, max_tokens 1), prefill_seconds from the reply: main 14.648 s mean, this PR 14.654 s.

• Tests. The full checklist suite (zig build test test-kernel-contract test-golden test-server-security test-pull-security fuzz-cortheon test-cuda-host) plus test-qwen35-formats-host: 478/481 steps succeeded. The one failure is keepalive_test "a silent engine's stream gets keepalives before its first token", which crashes the same way on main (a0d24896) on this machine. lean_new.py against main: 0 new problems. New host test: format coexistence: disjoint mixer and dense MLP scratch regions share one arena union (qwen35_formats_host.zig). It fails on main, where y and gate are separate rows.

• Not run. GB10 / sm_121. Metal (not affected). A MoE checkpoint served: its routed scratch still follows the shared region, but no MoE run was made.

Checklist

• Every token still goes through the lane rounds.
• Drafted output equals "draft": false, concurrent equals alone, and this branch's tokens equal main's.
• No precision traded for speed: no arithmetic changed.
• No GPU kernel changed.
• Prompt processing unchanged.
• Platforms: CUDA sm_86 only.
• New tests fail before the change, pass after it.
• Comments and docstrings are one line.
• No personal data, local paths or AI attribution lines.

[ashhart]

Thanks @diwakergupta. The union is careful, and we agree with your reading of it. We went through every user of the shared rows: the forward's mixers and dense MLP, the prompt attention and delta helpers, deltaScan, and the learning hooks, which copy on the engine's stream. Each mixer's rows are dead before the MLP writes gate. The errdefer at cuda_engine.zig:247 covers the pinned rows to the end of Engine.init. Merged onto current main, fmt, lean, test-qwen35-formats-host (your union test included), the EXL3 host steps, test-cuda-host and the CUDA cross-compile all pass.

1. Please rebase onto qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465 instead of qwen3_5 cuda: serve dense EXL3 checkpoints with the DFlash2 drafter #1474, which we're closing as a duplicate of qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465. Only the embedding lines in all() (cuda_exl3_weights.zig:236-244) overlap, so it should be small.
2. A note for the description: the mapped embedding applies to the MoE EXL3 checkpoints as well, since all() loads both kinds.

After the rebase we'll A/B the union on a GB10: the MLX 27B and a MoE MLX checkpoint against main (token hashes, concurrent == solo, drafted == plain, prefill and decode speed), and the dense EXL3 pack against #1465 by token hash. On a GB10 the mapped rows free nothing, because host and device share one pool, so there we'll only check that prefill isn't slower.

[ashhart]

Thanks @diwakergupta. The rebased commit is line for line the one we reviewed, and the description now says the mapped embedding covers the MoE EXL3 packs as well. Merged onto current main, fmt, lean (no new findings), test-qwen35-formats-host (10/10 with your union test), the EXL3 host steps, test-qwen35-draft-frame-host, test-cuda-host and the CUDA cross-compile all pass. It also merges cleanly with #1475 and #1507.

One small thing: the title still says "stacked on #1474"; please make it #1465.

Next we'll A/B it on a GB10 against main: the MLX 27B and a MoE MLX checkpoint (token hashes, concurrent == solo, drafted == plain, prefill and decode speed), and the dense EXL3 pack against #1465 by token hash, with prefill and decode speed, since every round's embedding rows now come from mapped host memory.

[diwakergupta]

Thanks @ashhart. The title now says #1465.

[ashhart]

Thanks @diwakergupta, nothing else is open here. After #1465 we'll A/B it on a GB10:

• the MLX 27B and a MoE MLX checkpoint against main: token hashes, concurrent against alone, drafted against plain, and
  prefill and decode speed;
• the dense EXL3 pack against qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts #1465 by token hash, with both speeds;
• a MoE EXL3 checkpoint against main, since its embedding moves to mapped host memory too.

We've started the CI runs that were waiting on our approval.

[diwakergupta]

Thanks @ashhart. One note on CI: zig-windows failed on rows close: a blocked spill-only flush cannot hold shutdown past its deadline (zig/src/core/rows_close_test.zig), a shutdown-deadline test this PR doesn't touch. The PR changes only four qwen3_5 CUDA files. It passes locally, and #1477's zig-windows passed on the same morning's runs, so it looks like a timing flake on the Windows runner. Could you rerun it?

[ashhart]

Thanks @diwakergupta, you're right. That's the Windows timing flake in rows_close_test, which main fixed in db3fd5ab after this run started. We've rerun the failed job.

[diwakergupta]

Hi @ashhart, checking in. The rerun of zig-windows failed again on the same rows_close_test deadline test. Reruns reuse the run's original merge commit, made on 2026-10-27 before db3fd5ab, so they don't pick up the fix. A new run on a fresh merge with main should pass. I can push a rebase if you'd like one. Also, is the GB10 A/B still waiting on #1465?

[ashhart]

Thanks @diwakergupta. We ran this on a GB10 (sm_121) against main, with Qwen3.8-27B EXL3 3.50bpw, fresh servers per arm and alternating base/new rounds.

The output is exact: 0 of 12 prompt-test rows differ from main. Speed came out slightly but consistently lower, though. Every candidate run in each of these cells was below every main run:

cell	main	this PR
1 stream	68.4	67.8 (-0.9%)
2 streams	107.4	106.4 (-0.9%)
4 streams	139.1	137.0 (-1.5%)
8 streams	168.9	165.3 (-2.1%)
decode after a 32,768-token prompt	38.1	37.4 (-1.8%)

Prefill is unchanged (within 0.6%). We don't land changes that slow the default path, so we're parking this for now. A likely suspect is the host-mapped embeddings, since every decode step would read them over the host link. If the shared mixer/MLP scratch on its own is neutral or faster, splitting that out would let us land it straight away.

[diwakergupta]

Thanks @ashhart. I split it as you suggested. Pushed f33ed2d2 on current main. It's the shared mixer/MLP scratch only, which frees 0.46 GiB on the 27B. On the 3090 Ti against main, alternating twice:

• 528/528 token hashes equal, with 0 unequal in concurrent vs alone and drafted vs plain.
• Decode is within ±0.1% in every cell where tokens per round match, at 1 and 4 streams. The cells that differ by up to 1.7% are the ones whose tokens per round moved between runs. The description has the table.
• Cold prefill: 14.648 s vs 14.654 s.

The host-mapped embedding is now #1678. It maps only when a discrete card has no room for the weights, one stream at the asked window and the drafter. On a 24 GB card that lets the 27B serve a 98k or 128k window that main refuses, and it never maps on unified memory like the GB10.

[ashhart]

Thanks @diwakergupta for splitting it. We ran the shared-scratch version (f33ed2d) on a GB10 against main, Qwen3.8-27B EXL3 3.50 bpw, fresh servers per arm, alternating base/new/new/base.

Tokens are equal to main (0 of 12 prompt-test rows differ), and almost everything is flat: 8k and 32k prompts within 0.3%, and 1, 2, 4 and 8 streams within 0.2%. One cell is consistently slower, though. Cold prefill on the 2,048-token prompts is 495.0 to 490.5 tok/s (-0.9%), and every run of this PR came in under every run of main:

run	2k prefill tok/s (four prompts)
main, 1st	496.2, 494.6, 495.8, 495.4
this PR, 1st	492.3, 489.7, 490.0, 488.7
this PR, 2nd	491.1, 491.9, 491.1, 487.9
main, 2nd	495.3, 493.5, 491.3, 492.7

Decode right after those prompts is -0.6% the same way. It only shows at the shortest prompt, so it looks like a fixed per-prompt cost rather than per-token work. A first touch of the shared region, or an extra clear or reservation when a prompt starts, would fit. We don't land a slower default path, so we're holding this for now. If that cost goes, the 0.46 GiB saving lands as it is, and we'll rerun the same cells.