Vulkan: PTQ1_0 decode is 6-119x slower than necessary (serial trit recurrence) + no MUL+FWHT fusion

[MrFadiAi]

Title: Vulkan: PTQ1_0 decode is 6-119x slower than necessary (serial trit recurrence) + no MUL+FWHT fusion

Hardware context: Radeon 890M iGPU (Ryzen AI 9 HX 470), Windows, Vulkan backend, Ternary-Bonsai-2-27B PTQ1_0 (5.95 GB), full offload, single-sequence decode.

Findings while profiling why Bonsai 2 PTQ1_0 decode runs at only ~2 t/s on this class of iGPU (Q2_0 band runs ~8 t/s on the same graph):

1. ptq1_0_trit() in vulkan-shaders/ptq1_0.glsl (and the duplicate in dequant_ptq1_0.comp) decodes each weight with a serial loop of up to 119 multiply-mod iterations (v = v*3 & 0xFF). This runs per weight per matvec. Since 3^64 == 1 (mod 256), the whole recurrence collapses to a 64-entry lookup table + one multiply — bit-identical, dramatically cheaper.

2. dequantize4() for PTQ1_0 calls ptq1_0_trit four times, re-evaluating the 3-way region branch per element. The codec layout guarantees any 4-aligned group shares one exponent and reads consecutive bytes, so one branch + a vector multiply decodes the group.

3. Metal has ggml_metal_op_can_fuse_fwht_signed (fusing the Hadamard sign MUL into the FWHT kernel, metal-ops.cpp ~2454). Vulkan has the same MUL -> RESHAPE -> FWHT-hinted MUL_MAT pattern in Bonsai 2's rotated-basis graphs but no fusion, so every Hadamard site pays an extra elementwise launch + full activation round-trip.

Proposed fix (implemented, tested locally): lookup-table trit decode + vectorized dequantize4 + MUL_FWHT fusion with dedicated fwht_signed_* pipelines, following the Metal implementation including the width-from-matmul validation (the activation may be a flat concat of n_blk transform-width rows, e.g. 5120 = 40 x 128).

Measured: PTQ1_0 decode 1.85 -> 3.26 t/s (+76%) on 890M; correctness 7/7 generation checks, bit-identical decode for the table/vectorization changes.

Happy to open the PR — wanted to file the issue first per CONTRIBUTING.md.

[blanham]

Corroboration from a discrete NVIDIA card, with a same-card control — and a data point that argues your fix is bigger than the 890M number suggests

Filed #201 before finding this issue; #201 is the same defect measured a different way and should probably be treated as corroboration of this one rather than as separate. Posting the numbers here since this is where the fix is.

Different hardware class entirely: RTX 2070 Max-Q (Turing, sm_75, discrete, ~256-bit GDDR6), driver 580.178.04, openSUSE Tumbleweed, same Ternary-Bonsai-2-27B-PTQ1_0.gguf, full offload.

llama-bench -ngl 99 -p 128 -n 32 -r 2:

model	backend	pp128 t/s	tg32 t/s
Bonsai 2 27B PTQ1_0	CUDA	185.62 ± 3.19	16.09 ± 0.01
Bonsai 2 27B PTQ1_0	Vulkan	110.90 ± 0.13	1.81 ± 0.02
Qwen2.5-7B Q4_K_M	CUDA	1252.45 ± 3.65	55.53 ± 0.16
Qwen2.5-7B Q4_K_M	Vulkan	1053.31 ± 195.26	44.60 ± 0.14

The control isolates it to the ternary path specifically

Vulkan ÷ CUDA generation, same card, same session:

• ordinary Q4_K_M: 0.80×
• ternary PTQ1_0: 0.11×

So the Vulkan backend is healthy on this card — 80% of CUDA on an ordinary quant. Prompt processing on PTQ1_0 is only 1.67× apart while generation is 8.9× apart, which is the split you would expect if the cost is per-weight in the matvec rather than anywhere in the batched path. That is consistent with the serial v = v*3 & 0xFF recurrence in ptq1_0_trit() being the dominant term.

PTQ1_0 on CUDA has nothing left to explain: 26.90 B params ÷ 7.62 B = 3.53× the compute per token, and 55.53 ÷ 3.53 = 15.7 against 16.09 measured.

The data point worth having

This issue measures 1.85 t/s on a Radeon 890M iGPU. This card measures 1.81 t/s.

A 890M integrated GPU and a discrete Turing card with several times the memory bandwidth produce the same decode rate, while the same card on CUDA does 16.09. The serial recurrence appears to dominate so completely that the hardware stops being the variable — which means the "6-119×" framing may be understating it for the discrete-GPU case, where there is much more hardware sitting idle behind the loop.

If useful, that also suggests a cheap regression check for the fix: after the lookup-table decode lands, PTQ1_0 decode should start scaling with the device again. Two very different GPUs converging on one number is itself the signature.

Happy to re-run this bench on the 2070 against your branch when the PR is up.

Scope

-r 2, short context, one machine, one session, 2026-09-18. The Q4_K_M control is one model at one quant, so "Vulkan is healthy on ordinary quants" is scoped to that row. I have not read the shader — the localisation above is by elimination across two models and two backends, and this issue's profiling is the actual diagnosis.

One incidental note for anyone comparing backends: llama-bench loads its ggml backend from the library path, so a Vulkan binary with a CUDA build's directory on LD_LIBRARY_PATH silently runs CUDA and reports backend CUDA. Also, on a laptop with an integrated GPU the discrete card may be Vulkan1 rather than Vulkan0 — this machine's iGPU runs the same model at 0.12 t/s, which is a very different number to accidentally report.

[bri-prism]

Thanks for the profiling. The mat-vec half of this is addressed by #238, which moves PTQ1_0 decode onto the integer-dot kernel from #188 and is in the new release prism-b10735-842b188. A rerun on your 890M with that build would be very useful.

The MUL+FWHT fusion is not part of that change and is still open in #187, so I'm leaving this issue open for that part.