qwen35: apply the Hadamard inverse to token embeddings in the MTP draft graph

[sudoingX]

Branch: pr-hadamard-mtp, one commit on top of prism (9a9394a89), file src/models/qwen35.cpp, 15 insertions.

What

llama_model_qwen35::graph_mtp looks the draft token's embedding up with a raw
ggml_get_rows(model.tok_embd, ...). When the model folds a Hadamard rotation into its weights
and lists token_embd.weight in prism.hadamard.inverse_weight_names (every Ternary Bonsai 2
file does), the rows of that table are stored rotated. The main graph restores the primal basis
right after the lookup in llm_graph_context::build_inp_embd(); the MTP graph did not. This
change applies the same llama_mul_mat_hadamard and sign vector after the lookup when the
table is in hadamard_inverses. Models without prism.hadamard.* keys take the path they took
before (the lookup finds nothing).

Why

No PrismML export carries an MTP block today, so the case never came up. It does the moment
someone grafts the Qwen 3.8 blk.64.nextn.* tensors onto Bonsai 2 to get --spec-type draft-mtp
(the graft tools and measurements are in github.com/sudoingX/bonsai2-small-gpu). Without the fix the
draft context is refused at load:

llama_verify_hadamard_graph: latent lookup 'mtp_tok_embd-64' consumed by op=RMS_NORM name='norm-64' src0 hint=0
llama_init_from_model: failed to initialize the context: Hadamard-latent table 'token_embd.weight' is read without the inverse transform
common_speculative_init_result: failed to create MTP context
srv    load_model: failed to create MTP context

The workaround is to ship a second, unrotated embedding table inside the head
(blk.64.nextn.embed_tokens.weight, a Q4_K copy of the donor's token_embd, 682 MiB of VRAM).
With the fix the head reads the trunk's own table and the copy is not needed: 9,956 MiB instead
of 10,638 MiB at 131072 context on an RTX 3060 12GB, and 196608 context fits (11,990 MiB) where
the fat file OOMs on the MTP compute buffer.

Reproduction

1. Ternary-Bonsai-2-27B-PTQ1_0.gguf plus the 15 blk.64.* tensors of any Qwen3.8-27B GGUF,
   with qwen35.block_count set to 65 and qwen35.nextn_predict_layers = 1
   (tools/extract_head.py --no-embed-tokens and tools/merge.py in the repo above; the
   merged file is 6,297,658,848 bytes, sha256 1e33c571...5685).
2. llama-server -m Ternary-Bonsai-2-27B-PTQ1_0-mtp-lean.gguf -ngl 99 -fa on -c 32768 -np 1 -ctk q4_0 -ctv q4_0 --jinja --spec-type draft-mtp --spec-draft-n-max 1
3. Before: the three lines above and exit. After: spec common_specu: adding speculative implementation 'draft-mtp', speculative decoding context initialized.

Measured (RTX 3060 12GB, 131072 context, q4_0 K/V, one slot)

	fat file, prebuilt prism-b10685	lean file, this fix
VRAM, n-max 1	10,638 MiB	9,956 MiB
draft acceptance, code / prose / bash (identity prompts)	0.91 / 0.65 / 0.80	0.89 / 0.67 / 0.77
greedy text with the flag	same three texts	same three texts
largest context with the head resident	163840	196608

The draft quality with the trunk's ternary, Hadamard-rotated embedding table equals the donor's
fp table: acceptance per run 0.85 to 0.94 on Python, 0.45 to 0.55 on prose, 0.73 to 0.79 on bash.

Known limits

• Only the qwen35 MTP graph is changed. Other architectures with an MTP graph
  (qwen3next, glm4-moe, deepseek2, ...) do the same raw lookup; none of them has a
  Hadamard-folded export today, so they are left alone here.
• A unit test would need a Hadamard-folded model with a nextn block; there is none in the tree.
  llama_verify_hadamard_graph is the guard that catches the bug, and it now passes on the
  file above.

[sudoingX]

Cross-reference: #210 adds llm_graph_context::build_embd_rows(), a ggml_get_rows followed by the inverse transform for Hadamard-latent tables, which is exactly the three ops this PR inlines in the qwen35 MTP graph. #210 does not touch src/models/qwen35.cpp, so the MTP draft graph still fails to build on Bonsai 2 after it.

If #210 lands first, this PR becomes a one-line change at qwen35.cpp:636, tok_embd = build_embd_rows(tok_embd_w, inp->tokens);, and the llama-impl.h include goes away. Ready to rebase to that form on request.

[bri-prism]

Agent review: posted by the maintainer's coding agent at their request.

No findings in the embedding inverse-transform change. The translation unit passes a Clang C++17 syntax check against same-base headers.

This is functionally the same fix as #205, with an additional include. Please consolidate the two PRs so only one implementation lands. This was a source/syntax review, not a model-backed MTP execution.

Reviewed commit: 1dc4a579b42905007dba3b3ef42e69fed57a9056.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot review overview

🟢 Approval recommended

The logic matches the established embedding path; only a non-blocking comment-format issue remains.

Review effort: Balanced
Findings: 1

Open (1)

• Condense oversized comment to repository style · New

[zhaoyilun]

Same fix as #205 — proposing we land one of them

Thanks for this, and for the cross-reference to #210 — that is the part I would have missed.

I compared the two patches line by line. The executable change is identical: the same find(tok_embd_w) on hadamard_inverses, the same llama_mul_mat_hadamard(ctx0, tok_embd, it->second.rot), the same conditional ggml_mul by it->second.signs, in the same position after the lookup. The only structural difference is the #include "llama-impl.h" here, and @bri-prism's review of #205 already established that translation unit compiles without it.

The maintainer has asked for one of the two to land. I have proposed #205 as the landing point, for the boring reason that it already carries two independent reproductions on unrelated stacks plus the Ada measurements, so no evidence has to be gathered again — not because the change here is any worse. It is not.

Your RTX 3060 numbers are better evidence than anything on #205 for the low-VRAM case, so I am folding them in there with attribution:

• 9,956 MiB instead of 10,638 MiB at 131072 context, and 196608 context fitting at 11,990 MiB where the fat file OOMs on the MTP compute buffer;
• acceptance unchanged (0.89 / 0.67 / 0.77 on code / prose / bash);
• your list of other architectures doing the same raw lookup (qwen3next, glm4-moe, deepseek2, …) — scoped out here, now recorded there as a known limit rather than a silent assumption;
• the build_embd_rows() rebase once dflash: apply the target's Hadamard transforms to borrowed embeddings and head #210 lands. I will take that on.

If you would rather #217 be the landing point, say so and I will move everything over instead. Absent that, I will treat #205 as the one and ask the maintainers to take it.

[sudoingX]

Agreed, land #205. It has the reproductions, and the change is the same three ops either way. I will close this one when #205 is in, or sooner if a maintainer prefers the queue clean. @renovys opened #230 this morning with the same fix, so that is a third copy worth folding into the same decision.

Thanks for carrying the numbers over. Two corrections so the ones on #205 are the current ones, both measured on an RTX 3060 12GB with the branch that carries this fix plus the PTQ1_0 mat-vec of #218, greedy, -ctk q4_0 -ctv q4_0 -np 1, three prompts:

• 131072 context: lean file 9,940 MiB, fat file 10,622 MiB, output byte identical between them.
• 196608 context: lean file 11,732 MiB. The fat file does not get there, it dies allocating the MTP compute buffer: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1040.28 MiB on device 0: cudaMalloc failed: out of memory.

That is the case for the fix in one line. Without it the only way to run --spec-type draft-mtp on a Hadamard-latent model is to ship a second, unrotated copy of the embedding table inside the GGUF, which costs 715 MB on disk and the deepest 65,536 tokens of context the card can otherwise hold.

Acceptance is unchanged by the fix, as expected, since it only corrects what the draft graph reads: 0.92 on Python, 0.79 on bash, 0.56 on prose at a fresh context, 0.65 to 0.73 on a long document between 18K and 120K tokens.

I have the card and both merged files here, so if #205 needs anything rechecked on 12GB, ask and I will run it.

[sudoingX]

#205 landed in prism (422590f, merge of zhaoyilun's mtp-hadamard-embedding) with the same change this pull request carried, the inverse Hadamard on the token embeddings the qwen35 MTP graph reads, and that was the agreed single landing point. Our bonsai2 branch now rebases straight onto prism without this commit, since the fix is in the tree. Thank you @zhaoyilun for carrying the 12GB evidence over and seeing it through; closing this one.

[Tongas]

Validated this on a real Hadamard-folded qwen35 with a grafted MTP head and it works — thanks for the fix. Cherry-picked 1dc4a57 onto b10687 (5d80cff) and ran Ternary-Bonsai-2-27B-Uncensored-Gaston (PQ2_0 and PTQ1_0) with a blk.64 nextn head grafted from a Qwen3.8-27B donor.
Before (stock branch): Hadamard-latent table 'token_embd.weight' is read without the inverse transform → failed to create MTP context, exactly as described.

After: the draft context builds and --spec-type draft-mtp runs. RTX 3090, -fa on, greedy, code prompt:

model	base	n-max 3	n-max 4	n-max 5
PQ2_0	67.2 t/s	84.5 (+26%)	88.0 (+31%)	74.2
PTQ1_0	56.3 t/s	—	55.8 (~flat)	—

n-max 4 is the sweet spot on PQ2_0 (+31% on code); n-max 5 regresses. On PTQ1_0 the grafted head didn't accelerate decode in my tests. Prose sees little/no gain either quant (low draft acceptance), high on code.

Also confirms the fix works for a grafted MTP head (donor embeddings), not just native. The model above is a public GGUF (I'm the author) if it's useful as a test artifact — it's an abliteration of Bonsai-2 with, incidentally, an honest-eval harness that measures why the usual refusal-removal scores come out inflated on reasoning models. Repro command happy to add if wanted.

Extra:

Posting the repro for the test-artifact record. Optimal --spec-draft-n-max looks hardware/workload dependent (lower values can win once concurrent load and GTT pressure are weighted), so treat these single-gen 3090 numbers as one regime, not universal. Build was prism b10687 (5d80cff) with 1dc4a57 cherry-picked, RTX 3090 24GB CUDA, single generation, greedy, code prompt:

MTP

llama-cli -m Ternary-Bonsai-2-27B-Uncensored-Gaston-PQ2_0-MTP.gguf
-ngl 99 -c 2048 -fa on --temp 0 -n 300 -st
--spec-type draft-mtp --spec-draft-n-max 4
-p $'<|im_start|>user\nWrite a Python function that implements binary search on a sorted list, with docstring and type hints. Then write 3 unit tests for it.<|im_end|>\n<|im_start|>assistant\n'

baseline: same command on the non-MTP file, drop --spec-type and --spec-draft-n-max

Numbers from the Generation: X t/s line, single-gen on the 3090: PQ2_0 base 67.2, n-max 3 84.5, n-max 4 88.0, n-max 5 74.2 t/s. PTQ1_0 was basically flat (~56 either way), prose prompts saw little to no gain.