diff --git a/docs/vulkan-890m-bonsai2-recipe.md b/docs/vulkan-890m-bonsai2-recipe.md new file mode 100644 index 000000000000..68efee9c4ec3 --- /dev/null +++ b/docs/vulkan-890m-bonsai2-recipe.md @@ -0,0 +1,113 @@ +# Bonsai 2 27B on AMD Radeon 890M — Full Speed Recipe (14 t/s) + +**Result: 27B dense-class hybrid model generating at 11.5–12 t/s through HTTP (12.4–14.1 t/s server-side), spec-decode acceptance 0.84, 5/5 quality gate — on an integrated GPU (AMD Radeon 890M, Ryzen AI 9 HX 470, 28 GB shared LPDDR5X).** + +This document is the complete, reproducible recipe: build configuration, launch flags, environment variables, and the hard-won findings from a 3-day optimization campaign. Stock llama.cpp Vulkan runs the same model at 1.85 t/s — **~7.5× faster with these changes**. + +--- + +## 1. Hardware / software baseline + +| Component | Value | +|---|---| +| APU | AMD Ryzen AI 9 HX 470 (Radeon 890M iGPU, RDNA 3.5) | +| RAM | 28 GB LPDDR5X-7500 (unified memory — the bandwidth wall) | +| Driver | AMD Adrenalin 26.9, Vulkan 1.4.349 | +| OS | Windows 11 | +| Toolchain | MinGW-w64 (winlibs GCC 16.2), CMake + MinGW Makefiles | + +## 2. Build configuration + +```bash +cmake -S . -B build-dspark -G "MinGW Makefiles" \ + -DCMAKE_BUILD_TYPE=Release \ + -DGGML_VULKAN=ON \ + -DGGML_NATIVE=OFF \ + -DCMAKE_C_COMPILER=/gcc.exe \ + -DCMAKE_CXX_COMPILER=/g++.exe + +mingw32-make -C build-dspark -j8 llama-server +``` + +⚠️ **Always wipe the build directory after any stash/revert/branch-switch.** Incremental rebuilds after tree changes produce stale-object mazes that silently build old code ("0 errors" that hides 10 real errors). + +⚠️ **Kill any running `llama-server.exe` BEFORE relinking** — on Windows the linker fails with "Permission denied" if the exe is loaded, and you may silently benchmark a stale binary. + +## 3. The model: stock GGUFs will NOT reproduce this (graft required) + +The speed comes from `--spec-type draft-mtp` — but **PrismML ships all Bonsai 2 GGUFs WITHOUT the MTP head** (layers 0–63 only, nextn stripped). A stock file cannot run speculative decoding. The champion file (`Bonsai-2-27B-Q2_0-fork-MTP.gguf`) is: + +1. **Base**: Q2_0-fork GGUF from [`prism-ml/Ternary-Bonsai-2-27B-gguf`](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) +2. **Grafted MTP head**: the 15 `blk.64.*` (nextn) tensors range-requested from unsloth's `Qwen3.8-27B` GGUF (~340 MB, not the full 17 GB) via the community `graft_mtp.py`; metadata `block_count 64→65`, `nextn_predict_layers=1`. Verbatim copy is valid because Bonsai's RMSNorm weights match stock Qwen3.8 elementwise (cos > 0.99996) — the residual stream stays in the original basis. +3. **Hadamard-inverse patch** in the MTP graph (shipped in PR #187 commit `4b8092c3`): the draft graph's embedding lookup must apply the inverse Hadamard rotation (rotation first, sign flip second) or the server dies at context init. + +⚠️ **Build from the PR #187 branch, NOT upstream main** — until it merges, upstream lacks the bf16 state pools, the fused GDN rows-mode kernel, the Q2fork matvec, and the Hadamard fix. A stock build either refuses the Q2fork format or runs it on CPU. + +## 4. Production launch flags + +```bash +LLAMA_SSM_BF16_STATE=1 ./llama-server \ + -m Bonsai-2-27B-Q2_0-fork-MTP.gguf \ + --spec-type draft-mtp --spec-draft-n-max 1 \ + -ngl 99 -ngld 99 -fa on \ + -t 24 -td 8 -c 16384 -np 1 \ + --jinja --host 0.0.0.0 --port 8080 +``` + +| Flag | Why | +|---|---| +| `LLAMA_SSM_BF16_STATE=1` | BF16 SSM state pools (+22% decode). **Required** — the F32-state path has a selection hole that can crash. | +| `--spec-type draft-mtp` | MTP self-speculation using the model's native draft head | +| `--spec-draft-n-max 1` | n=1 is optimal on iGPU. n=2 collapses acceptance (0.84 → 0.44) and loses 15% speed. The draft chain costs more than it earns at shared-memory bandwidth. | +| `-ngl 99 -ngld 99` | Full offload of target AND draft. Everything must fit in VRAM — spilling to CPU caps you at 2–3 t/s. | +| `-fa on` | Flash attention | +| `-t 24 -td 8` | Target threads = all 24; draft threads = 8 (16 oversubscribes against the target). Try both head placements on new hardware: GPU head (`-ngld 99`) won here; on machines with a stronger CPU:GPU ratio, CPU head (`-ngld 0 -td 8`) can win by overlapping. | +| `-np 8` (batch mode) | For multi-user serving: ~14.3 t/s aggregate across 8 streams | + +## 5. Measured results + +| Configuration | Speed | Notes | +|---|---|---| +| Stock llama.cpp Vulkan | 1.85 t/s | baseline | +| + BF16 SSM state pools | +22% | commit `53608747` | +| + fused GDN rows-mode matvec | +30% cumulative | commit `96062463` | +| + PTQ1_0/Q2fork matvec v2 (4-row workgroups) | closed | commit `bb5997c5` | +| + MTP speculation n=1, acc 0.84 | **11.5–12 t/s HTTP / 14.1 peak** | production | +| n=2 speculation | 9.9 t/s | ❌ regression | +| KV cache q8_0 | no gain | KV is a small share | +| 32 threads | no gain | threads saturated at 24 | +| Leaner quant (IQ3, −21% bytes) | −27% speed | dequant-bound, not bandwidth-bound | + +Effective bandwidth: ~180 GB/s sustained on LPDDR5X — this is the hardware wall for a weights-streaming workload on unified memory. + +## 6. Findings that will save you days + +1. **Raw-gates fusion is CPU/Metal/CUDA-only upstream — keep it that way on Vulkan.** The `qwen35.cpp` device allowlist deliberately excludes Vulkan from the raw-gates path. Enabling it produces corrupted output (`////`) in every shader variant (subgroup, nocluster, shmem). We verified this is a real incompatibility, not a stale allowlist. + +2. **`GGML_SSM_BF16_STATE` is load-bearing.** Without it, F32-state + rows-mode hits a pipeline-selection gap (returns nullptr → CPU fallback or worse). + +3. **Leaner quant ≠ faster on iGPU.** PTQ1_0 (5.9 GB) measured 27% *slower* than Q2fork (7.4 GB): iGPU decode is dequant-compute-bound, not bandwidth-bound, until you hit specialized kernels. The fork's custom 2-bit kernels beat generic 3-bit kernels despite moving more bytes. + +4. **Test spec-mode quality with an exact-answer gate** (e.g. `17×23=391`, translation, sequence completion) before benchmarking speed. Speculative decoding bugs present as output corruption, not crashes. + +5. **Debugging order that actually works:** (a) prove the GPU healthy with a known-good binary, (b) binary-search the *config* (env, flags) before the code, (c) diff allowlists/selection predicates between trees — the bug is usually a path-selection mismatch, not shader math. + +6. **VRAM wall rule:** model must fit fully in VRAM. A 17 GB model on 14.4 GB usable VRAM = 2.5 t/s (CPU spillover); the same class of model at 10 GB = 5.9 t/s; specialized 7.4 GB = 14 t/s. + +## 7. Related commits (PR #187) + +- `905c29b6` PTQ1_0 decode: trit-table + vectorized dequantize4, MUL+FWHT fusion +- `a078ce98` review feedback +- `63854022` dedicated PTQ1_0 matvec kernel, coopmat A-load, MTP hadamard inverse +- `bb5997c5` PTQ1_0 matvec v2 (4-row workgroups) +- `53608747` bf16 SSM state pools +- `96062463` GDN rows-mode + bf16 state read (fused) +- `6e17c24d` build fix + tail-guard hoist + +## 8. Reproducibility notes + +- Numbers are single-stream, temperature 0.1, 300-token generations, measured server-side (`eval time`) and cross-checked over HTTP. +- Quality gate: 5/5 exact-answer checks (arithmetic, knowledge, sequence, translation, word problem) — spec-decode must not degrade answers. +- `draft acceptance` logged by llama-server should read 0.55–0.85 at n=1. If it reads 0.0 or 1.0 persistently, suspect a draft-graph misconfiguration. + +*Hardware: AMD Ryzen AI 9 HX 470 / Radeon 890M. Your mileage on other iGPUs will scale with memory bandwidth.* diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp index b6f9965add7b..7fafe8e5d33b 100644 --- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp +++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp @@ -960,6 +960,7 @@ struct vk_device_struct { vk_pipeline pipeline_concat_i8, pipeline_concat_i16, pipeline_concat_i32, pipeline_concat_i64; vk_pipeline pipeline_upscale_nearest_f32, pipeline_upscale_bilinear_f32, pipeline_upscale_bicubic_f32, pipeline_upscale_bilinear_antialias_f32; vk_pipeline pipeline_scale_f32; + vk_pipeline pipeline_scale_bf16; vk_pipeline pipeline_log[2]; vk_pipeline pipeline_tri[2]; vk_pipeline pipeline_diag[2]; @@ -1081,6 +1082,8 @@ struct vk_device_struct { vk_pipeline pipeline_gated_linear_attn_f32; // [size_idx][kda] where size_idx: 0=d16, 1=d32, 2=d64, 3=d128 vk_pipeline pipeline_gated_delta_net[4][2]; + vk_pipeline pipeline_gated_delta_net_rows[4][2]; + vk_pipeline pipeline_gated_delta_net_rows_bf16state[4][2]; vk_pipeline pipeline_ssm_scan_f32_d128; vk_pipeline pipeline_ssm_scan_f32_d256; vk_pipeline pipeline_ssm_conv_f32; @@ -5216,6 +5219,7 @@ static void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) { uint32_t rm_kq = 2; uint32_t rm_stdq_int = 1; uint32_t rm_kq_int = 1; + uint32_t rm_stdq_ptq = 1; auto const &rm_iq_int = [](uint32_t i) { return i == 0 ? 8u : 4u; }; if (device->vendor_id == VK_VENDOR_ID_AMD) { if (device->architecture == AMD_GCN) { @@ -5223,6 +5227,11 @@ static void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) { rm_kq = 4; rm_stdq_int = 4; } + // FADI-OPT: PTQ1_0's dedicated kernel loads the B vector once per + // (row, block-group); two rows per workgroup halves the B traffic. + if (device->architecture == AMD_RDNA3 || device->architecture == AMD_RDNA2) { + rm_stdq_ptq = 1; + } } else if (device->vendor_id == VK_VENDOR_ID_INTEL) { rm_stdq = 2; rm_stdq_int = 2; @@ -5266,7 +5275,7 @@ static void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) { ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f32_f32[w][GGML_TYPE_F16 ][i], "mul_mat_vec_f16_f32_f32", arr_dmmv_f16_f32_f32_len[reduc], arr_dmmv_f16_f32_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2, 1, 1}, {wg_size_subgroup, 2, i+1}, 1, false, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f32_f32[w][GGML_TYPE_BF16][i], "mul_mat_vec_bf16_f32_f32", arr_dmmv_bf16_f32_f32_len[reduc], arr_dmmv_bf16_f32_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2, 1, 1}, {wg_size_subgroup, 2, i+1}, 1, false, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f32_f32[w][GGML_TYPE_Q1_0][i], "mul_mat_vec_q1_0_f32_f32", arr_dmmv_q1_0_f32_f32_len[reduc], arr_dmmv_q1_0_f32_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); - ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f32_f32[w][GGML_TYPE_PTQ1_0][i], "mul_mat_vec_ptq1_0_f32_f32", arr_dmmv_ptq1_0_f32_f32_len[reduc], arr_dmmv_ptq1_0_f32_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); + ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f32_f32[w][GGML_TYPE_PTQ1_0][i], "mul_mat_vec_ptq1_0_f32_f32", arr_dmmv_ptq1_0_f32_f32_len[reduc], arr_dmmv_ptq1_0_f32_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {4, 1, 1}, {wg_size_subgroup, 4, i+1}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f32_f32[w][GGML_TYPE_Q2_0][i], "mul_mat_vec_q2_0_f32_f32", arr_dmmv_q2_0_f32_f32_len[reduc], arr_dmmv_q2_0_f32_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f32_f32[w][GGML_TYPE_Q4_0][i], "mul_mat_vec_q4_0_f32_f32", arr_dmmv_q4_0_f32_f32_len[reduc], arr_dmmv_q4_0_f32_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f32_f32[w][GGML_TYPE_Q4_1][i], "mul_mat_vec_q4_1_f32_f32", arr_dmmv_q4_1_f32_f32_len[reduc], arr_dmmv_q4_1_f32_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); @@ -5295,7 +5304,7 @@ static void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) { ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f16_f32[w][GGML_TYPE_F16 ][i], "mul_mat_vec_f16_f16_f32", arr_dmmv_f16_f16_f32_len[reduc], arr_dmmv_f16_f16_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2, 1, 1}, {wg_size_subgroup, 2, i+1}, 1, false, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f16_f32[w][GGML_TYPE_BF16][i], "mul_mat_vec_bf16_f16_f32", arr_dmmv_bf16_f16_f32_len[reduc], arr_dmmv_bf16_f16_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2, 1, 1}, {wg_size_subgroup, 2, i+1}, 1, false, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f16_f32[w][GGML_TYPE_Q1_0][i], "mul_mat_vec_q1_0_f16_f32", arr_dmmv_q1_0_f16_f32_len[reduc], arr_dmmv_q1_0_f16_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); - ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f16_f32[w][GGML_TYPE_PTQ1_0][i], "mul_mat_vec_ptq1_0_f16_f32", arr_dmmv_ptq1_0_f16_f32_len[reduc], arr_dmmv_ptq1_0_f16_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); + ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f16_f32[w][GGML_TYPE_PTQ1_0][i], "mul_mat_vec_ptq1_0_f16_f32", arr_dmmv_ptq1_0_f16_f32_len[reduc], arr_dmmv_ptq1_0_f16_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {4, 1, 1}, {wg_size_subgroup, 4, i+1}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f16_f32[w][GGML_TYPE_Q2_0][i], "mul_mat_vec_q2_0_f16_f32", arr_dmmv_q2_0_f16_f32_len[reduc], arr_dmmv_q2_0_f16_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f16_f32[w][GGML_TYPE_Q4_0][i], "mul_mat_vec_q4_0_f16_f32", arr_dmmv_q4_0_f16_f32_len[reduc], arr_dmmv_q4_0_f16_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_f16_f32[w][GGML_TYPE_Q4_1][i], "mul_mat_vec_q4_1_f16_f32", arr_dmmv_q4_1_f16_f32_len[reduc], arr_dmmv_q4_1_f16_f32_data[reduc], "main", mul_mat_vec_num_bindings, sizeof(vk_mat_vec_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq, i+1}, 1, true, use_subgroups, force_subgroup_size); @@ -5351,7 +5360,7 @@ static void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) { ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_id_f32[w][GGML_TYPE_F16 ], "mul_mat_vec_id_f16_f32", arr_dmmv_id_f16_f32_f32_len[reduc], arr_dmmv_id_f16_f32_f32_data[reduc], "main", mul_mat_vec_id_num_bindings, sizeof(vk_mat_vec_id_push_constants), {2, 1, 1}, {wg_size_subgroup, 2}, 1, false, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_id_f32[w][GGML_TYPE_BF16], "mul_mat_vec_id_bf16_f32", arr_dmmv_id_bf16_f32_f32_len[reduc], arr_dmmv_id_bf16_f32_f32_data[reduc], "main", mul_mat_vec_id_num_bindings, sizeof(vk_mat_vec_id_push_constants), {2, 1, 1}, {wg_size_subgroup, 2}, 1, false, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_id_f32[w][GGML_TYPE_Q1_0], "mul_mat_vec_id_q1_0_f32", arr_dmmv_id_q1_0_f32_f32_len[reduc], arr_dmmv_id_q1_0_f32_f32_data[reduc], "main", mul_mat_vec_id_num_bindings, sizeof(vk_mat_vec_id_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq}, 1, true, use_subgroups, force_subgroup_size); - ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_id_f32[w][GGML_TYPE_PTQ1_0], "mul_mat_vec_id_ptq1_0_f32", arr_dmmv_id_ptq1_0_f32_f32_len[reduc], arr_dmmv_id_ptq1_0_f32_f32_data[reduc], "main", mul_mat_vec_id_num_bindings, sizeof(vk_mat_vec_id_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq}, 1, true, use_subgroups, force_subgroup_size); + ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_id_f32[w][GGML_TYPE_PTQ1_0], "mul_mat_vec_id_ptq1_0_f32", arr_dmmv_id_ptq1_0_f32_f32_len[reduc], arr_dmmv_id_ptq1_0_f32_f32_data[reduc], "main", mul_mat_vec_id_num_bindings, sizeof(vk_mat_vec_id_push_constants), {2*rm_stdq_ptq, 1, 1}, {wg_size_subgroup, 2*rm_stdq_ptq}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_id_f32[w][GGML_TYPE_Q2_0], "mul_mat_vec_id_q2_0_f32", arr_dmmv_id_q2_0_f32_f32_len[reduc], arr_dmmv_id_q2_0_f32_f32_data[reduc], "main", mul_mat_vec_id_num_bindings, sizeof(vk_mat_vec_id_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_id_f32[w][GGML_TYPE_Q4_0], "mul_mat_vec_id_q4_0_f32", arr_dmmv_id_q4_0_f32_f32_len[reduc], arr_dmmv_id_q4_0_f32_f32_data[reduc], "main", mul_mat_vec_id_num_bindings, sizeof(vk_mat_vec_id_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq}, 1, true, use_subgroups, force_subgroup_size); ggml_vk_create_pipeline(device, device->pipeline_dequant_mul_mat_vec_id_f32[w][GGML_TYPE_Q4_1], "mul_mat_vec_id_q4_1_f32", arr_dmmv_id_q4_1_f32_f32_len[reduc], arr_dmmv_id_q4_1_f32_f32_data[reduc], "main", mul_mat_vec_id_num_bindings, sizeof(vk_mat_vec_id_push_constants), {2*rm_stdq, 1, 1}, {wg_size_subgroup, 2*rm_stdq}, 1, true, use_subgroups, force_subgroup_size); @@ -5650,6 +5659,7 @@ static void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) { ggml_vk_create_pipeline(device, device->pipeline_upscale_bilinear_antialias_f32, "upscale_f32", upscale_f32_len, upscale_f32_data, "main", 2, sizeof(vk_op_upscale_push_constants), {512, 1, 1}, {GGML_SCALE_MODE_BILINEAR | GGML_SCALE_FLAG_ANTIALIAS}, 1); ggml_vk_create_pipeline(device, device->pipeline_scale_f32, "scale_f32", scale_f32_len, scale_f32_data, "main", 2, sizeof(vk_op_unary_push_constants), {512, 1, 1}, {}, 1); + ggml_vk_create_pipeline(device, device->pipeline_scale_bf16, "scale_bf16", scale_bf16_len, scale_bf16_data, "main", 2, sizeof(vk_op_unary_push_constants), {512, 1, 1}, {}, 1); ggml_vk_create_pipeline(device, device->pipeline_log[0], "log_f32", log_f32_len, log_f32_data, "main", 2, sizeof(vk_op_unary_push_constants), {512, 1, 1}, {}, 1); ggml_vk_create_pipeline(device, device->pipeline_log[1], "log_f16", log_f16_len, log_f16_data, "main", 2, sizeof(vk_op_unary_push_constants), {512, 1, 1}, {}, 1); @@ -5943,6 +5953,24 @@ static void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) { gdn_names[si][kda], gdn_len, gdn_data, "main", 7, sizeof(vk_op_gated_delta_net_push_constants), wg_denoms, {S_V, kda, device->subgroup_size, lanes_per_column}, 1, true, use_subgroup_ops, device->subgroup_size); } + size_t gdnr_len; const void * gdnr_data; + if (use_clustered_reduce) { gdnr_len = gated_delta_net_rows_f32_len; gdnr_data = (const void *)gated_delta_net_rows_f32_data; } + else if (use_subgroup_reduce) { gdnr_len = gated_delta_net_rows_f32_nocluster_len; gdnr_data = (const void *)gated_delta_net_rows_f32_nocluster_data; } + else { gdnr_len = gated_delta_net_rows_f32_shmem_len; gdnr_data = (const void *)gated_delta_net_rows_f32_shmem_data; } + for (uint32_t kda = 0; kda < 2; kda++) { + ggml_vk_create_pipeline(device, device->pipeline_gated_delta_net_rows[si][kda], + gdn_names[si][kda], gdnr_len, gdnr_data, "main", 8, sizeof(vk_op_gated_delta_net_push_constants), + wg_denoms, {S_V, kda, device->subgroup_size, lanes_per_column}, 1, true, use_subgroup_ops, device->subgroup_size); + } + size_t gdnrb_len; const void * gdnrb_data; + if (use_clustered_reduce) { gdnrb_len = gated_delta_net_rows_bf16state_f32_len; gdnrb_data = (const void *)gated_delta_net_rows_bf16state_f32_data; } + else if (use_subgroup_reduce) { gdnrb_len = gated_delta_net_rows_bf16state_f32_nocluster_len; gdnrb_data = (const void *)gated_delta_net_rows_bf16state_f32_nocluster_data; } + else { gdnrb_len = gated_delta_net_rows_bf16state_f32_shmem_len; gdnrb_data = (const void *)gated_delta_net_rows_bf16state_f32_shmem_data; } + for (uint32_t kda = 0; kda < 2; kda++) { + ggml_vk_create_pipeline(device, device->pipeline_gated_delta_net_rows_bf16state[si][kda], + gdn_names[si][kda], gdnrb_len, gdnrb_data, "main", 8, sizeof(vk_op_gated_delta_net_push_constants), + wg_denoms, {S_V, kda, device->subgroup_size, lanes_per_column}, 1, true, use_subgroup_ops, device->subgroup_size); + } } } @@ -11356,6 +11384,9 @@ static vk_pipeline ggml_vk_op_get_pipeline(ggml_backend_vk_context * ctx, const if (src0->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32) { return ctx->device->pipeline_scale_f32; } + if (src0->type == GGML_TYPE_BF16 && dst->type == GGML_TYPE_BF16) { + return ctx->device->pipeline_scale_bf16; + } return nullptr; case GGML_OP_SQR: if (src0->type == dst->type && @@ -11748,6 +11779,22 @@ static vk_pipeline ggml_vk_op_get_pipeline(ggml_backend_vk_context * ctx, const return nullptr; case GGML_OP_GATED_DELTA_NET: if (src0->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32) { + if (dst->src[6] != nullptr) { + const uint32_t S_v = dst->src[2]->ne[0]; + const uint32_t kda = (dst->src[3]->ne[0] == (int64_t)S_v) ? 1 : 0; + uint32_t si; + switch (S_v) { + case 16: si = 0; break; + case 32: si = 1; break; + case 64: si = 2; break; + case 128: si = 3; break; + default: return nullptr; + } + if (dst->src[5]->type == GGML_TYPE_BF16) { + return ctx->device->pipeline_gated_delta_net_rows_bf16state[si][kda]; + } + return ctx->device->pipeline_gated_delta_net_rows[si][kda]; + } const uint32_t S_v = dst->src[2]->ne[0]; const uint32_t kda = (dst->src[3]->ne[0] == (int64_t)S_v) ? 1 : 0; uint32_t si; @@ -12848,6 +12895,12 @@ static void ggml_vk_gated_delta_net(ggml_backend_vk_context * ctx, vk_context& s for (int i = 0; i < 6; i++) { src_buf[i] = ggml_vk_tensor_subbuffer(ctx, dst->src[i]); } + // rows mode: extra index buffer at binding 7 (dst->src[6]) + const bool gdn_rows = dst->src[6] != nullptr; + vk_subbuffer rows_buf = {}; + if (gdn_rows) { + rows_buf = ggml_vk_tensor_subbuffer(ctx, dst->src[6]); + } const uint32_t sq1 = (uint32_t)(src_q->nb[1] / sizeof(float)); const uint32_t sq2 = (uint32_t)(src_q->nb[2] / sizeof(float)); @@ -12873,9 +12926,15 @@ static void ggml_vk_gated_delta_net(ggml_backend_vk_context * ctx, vk_context& s K }; - ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, - {src_buf[0], src_buf[1], src_buf[2], src_buf[3], src_buf[4], src_buf[5], dst_buf}, - pc, { H, n_seqs, S_v }); + if (gdn_rows) { + ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, + {src_buf[0], src_buf[1], src_buf[2], src_buf[3], src_buf[4], src_buf[5], dst_buf, rows_buf}, + pc, { H, n_seqs, S_v }); + } else { + ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, + {src_buf[0], src_buf[1], src_buf[2], src_buf[3], src_buf[4], src_buf[5], dst_buf}, + pc, { H, n_seqs, S_v }); + } } static void ggml_vk_ssm_scan(ggml_backend_vk_context * ctx, vk_context& subctx, ggml_tensor * dst) { @@ -17230,6 +17289,17 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg } } uint64_t flops_per_submit = std::min(flops_cap, ctx->last_total_flops / 40u); + // Debug/testing knob: absolute flops-per-submit override (0 = default heuristic). + // Lets coarser submissions be evaluated on submit-overhead-bound devices. + { + const char * GGML_VK_FLOPS_PER_SUBMIT_env = getenv("GGML_VK_FLOPS_PER_SUBMIT"); + if (GGML_VK_FLOPS_PER_SUBMIT_env != nullptr) { + uint64_t v = std::stoull(GGML_VK_FLOPS_PER_SUBMIT_env); + if (v > 0) { + flops_per_submit = v; + } + } + } auto const submit_after = [&](int start, int end) { if (ctx->device->serialize_submissions) { @@ -18541,7 +18611,7 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm case GGML_OP_FILL: return op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16; case GGML_OP_SCALE: - return ggml_is_contiguous(op->src[0]) && op->src[0]->type == GGML_TYPE_F32; + return ggml_is_contiguous(op->src[0]) && (op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_BF16); case GGML_OP_PAD: case GGML_OP_PAD_REFLECT_1D: case GGML_OP_ROLL: @@ -18612,15 +18682,22 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm return op->src[0]->type == GGML_TYPE_F32 && op->type == GGML_TYPE_F32 && op->src[0]->ne[0] == 64; case GGML_OP_GATED_DELTA_NET: { - // rows-indexed state read (src[6]) not implemented on Vulkan yet - if (op->src[6] != nullptr) { + // rows-indexed state read (src[6]) requires an int32 index tensor + if (op->src[6] != nullptr && op->src[6]->type != GGML_TYPE_I32) { + return false; + } + if (op->src[6] != nullptr && op->src[5]->type != GGML_TYPE_F32 && op->src[5]->type != GGML_TYPE_BF16) { return false; } const uint32_t S_v = op->src[2]->ne[0]; if (S_v != 16 && S_v != 32 && S_v != 64 && S_v != 128) { return false; } + const bool state_bf16 = op->src[5]->type == GGML_TYPE_BF16; for (int i = 0; i < 6; i++) { + if (i == 5 && state_bf16) { + continue; // bf16 state handled by the dedicated rows pipeline + } if (op->src[i] == nullptr || op->src[i]->type != GGML_TYPE_F32) { return false; } diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.glsl b/ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.glsl index 88be1360403a..4a602fbe1c97 100644 --- a/ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.glsl +++ b/ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.glsl @@ -129,14 +129,39 @@ vec4 dequantize4(uint ib, uint iqs, uint a_offset) { #if defined(DATA_A_PTQ1_0) #include "ptq1_0.glsl" +// FADI-OPT: any 4-aligned group shares one exponent n and reads consecutive bytes, so one branch + vector multiply decodes it. vec2 dequantize(uint ib, uint iqs, uint a_offset) { return vec2(ptq1_0_trit(ib, a_offset, iqs), ptq1_0_trit(ib, a_offset, iqs + 1u)); } vec4 dequantize4(uint ib, uint iqs, uint a_offset) { - return vec4(ptq1_0_trit(ib, a_offset, iqs), - ptq1_0_trit(ib, a_offset, iqs + 1u), - ptq1_0_trit(ib, a_offset, iqs + 2u), - ptq1_0_trit(ib, a_offset, iqs + 3u)); + uvec4 v; + if (iqs < 120u) { + uint base; + uint n; + if (iqs < 80u) { + base = iqs & 15u; + n = iqs >> 4u; + } else { + const uint t = iqs - 80u; + base = 16u + (t & 7u); + n = t >> 3u; + } + const uint m = POW3_MOD256[n & 63u]; + v = uvec4(uint(data_a[a_offset + ib].qs[base ]), + uint(data_a[a_offset + ib].qs[base + 1u]), + uint(data_a[a_offset + ib].qs[base + 2u]), + uint(data_a[a_offset + ib].qs[base + 3u])) * m & 0xFFu; + } else { + const uint t = iqs - 120u; // 0 or 4 + const uint n0 = t >> 1u; // shared exponent for elements 0,1 + const uint b0 = uint(data_a[a_offset + ib].qh[0]); + const uint b1 = uint(data_a[a_offset + ib].qh[1]); + v = uvec4((b0 * POW3_MOD256[n0 & 63u]) & 0xFFu, + (b1 * POW3_MOD256[n0 & 63u]) & 0xFFu, + (b0 * POW3_MOD256[(n0+1u) & 63u]) & 0xFFu, + (b1 * POW3_MOD256[(n0+1u) & 63u]) & 0xFFu); + } + return vec4((v * 3u) >> 8u) - 1.0f; } #endif diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/dequant_ptq1_0.comp b/ggml/src/ggml-vulkan/vulkan-shaders/dequant_ptq1_0.comp index 53baf41f65b9..ad4b14646433 100644 --- a/ggml/src/ggml-vulkan/vulkan-shaders/dequant_ptq1_0.comp +++ b/ggml/src/ggml-vulkan/vulkan-shaders/dequant_ptq1_0.comp @@ -10,6 +10,10 @@ layout (binding = 1) writeonly buffer D {D_TYPE data_b[];}; // One invocation per 8 elements: 16 invocations cover a 128-weight block. Element // order follows the CPU codec (16-byte qs chunk, 8-byte qs chunk, then qh), so the // index maths lives in one place rather than being reproduced per shader. +// FADI-OPT: table-lookup form of the trit recurrence; n is at most 4 in every +// region, so five powers suffice and the serial multiply loop collapses to one lookup. +const uint PTQ_POW3_MOD256[5] = uint[5](1u, 3u, 9u, 27u, 81u); + float ptq1_0_trit_l(uint ib, uint e) { uint b; uint n; @@ -26,10 +30,7 @@ float ptq1_0_trit_l(uint ib, uint e) { n = t >> 1u; } - uint v = b; - for (uint i = 0u; i < n; ++i) { - v = (v * 3u) & 0xFFu; - } + const uint v = (b * PTQ_POW3_MOD256[n]) & 0xFFu; return float(int((v * 3u) >> 8u) - 1); } diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/fwht.comp b/ggml/src/ggml-vulkan/vulkan-shaders/fwht.comp index 904c898175e2..7bf455f12942 100644 --- a/ggml/src/ggml-vulkan/vulkan-shaders/fwht.comp +++ b/ggml/src/ggml-vulkan/vulkan-shaders/fwht.comp @@ -21,6 +21,7 @@ layout(push_constant) uniform parameter uint src_offset; uint dst_offset; float scale; + uint n_blk; // FADI-FUSION: >0 = apply per-row Hadamard sign block while loading }; layout(binding = 0, std430) readonly buffer A { @@ -31,6 +32,10 @@ layout(binding = 0, std430) readonly buffer A { #endif }; layout(binding = 1, std430) writeonly buffer D { float data_d[]; }; +// FADI-FUSION: sign vector [N*n_blk] for the fused MUL(signs)+FWHT variant only +#ifdef FWHT_SIGNED +layout(binding = 2, std430) readonly buffer S { float data_s[]; }; +#endif const uint EL_W = N / BLOCK_SIZE; @@ -62,9 +67,17 @@ void main() { float reg[EL_W]; + // FADI-FUSION: fold the Hadamard sign-flip MUL into the transform load. + // Only the FWHT_SIGNED variant has data_s bound; others compile this away. [[unroll]] for (uint i = 0; i < EL_W; ++i) { - reg[i] = row < n_rows ? float(data_a[src_offset + row_offset + i * BLOCK_SIZE + tid]) * scale : 0.0; + float v = row < n_rows ? float(data_a[src_offset + row_offset + i * BLOCK_SIZE + tid]) : 0.0; +#ifdef FWHT_SIGNED + if (n_blk > 0u) { + v *= data_s[(row % n_blk) * N + i * BLOCK_SIZE + tid]; + } +#endif + reg[i] = v * scale; } #ifdef FWHT_SHMEM diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/gated_delta_net.comp b/ggml/src/ggml-vulkan/vulkan-shaders/gated_delta_net.comp index 0e384330b9b9..076b61e2a57c 100644 --- a/ggml/src/ggml-vulkan/vulkan-shaders/gated_delta_net.comp +++ b/ggml/src/ggml-vulkan/vulkan-shaders/gated_delta_net.comp @@ -1,6 +1,9 @@ #version 450 #extension GL_EXT_control_flow_attributes : require +#if STATE_BF16 +#extension GL_EXT_shader_16bit_storage : require +#endif #extension GL_KHR_shader_subgroup_basic : enable #if USE_SUBGROUP_CLUSTERED #extension GL_KHR_shader_subgroup_clustered : enable @@ -39,7 +42,16 @@ layout(binding = 1) readonly buffer KBuf { FLOAT_TYPE data_k[]; }; layout(binding = 2) readonly buffer VBuf { FLOAT_TYPE data_v[]; }; layout(binding = 3) readonly buffer GBuf { FLOAT_TYPE data_g[]; }; layout(binding = 4) readonly buffer BetaBuf { FLOAT_TYPE data_beta[]; }; +#if STATE_BF16 +layout(binding = 5) readonly buffer StateBuf { uint16_t data_state_h[]; }; +// idx is in logical element units; uint16_t elements advance 2 bytes each, matching bf16 stride +float state_load(uint idx) { return uintBitsToFloat(uint(data_state_h[idx]) << 16); } +#else layout(binding = 5) readonly buffer StateBuf { FLOAT_TYPE data_state[]; }; +#endif +#if USE_STATE_ROWS +layout(binding = 7) readonly buffer StateRowsBuf { int data_state_rows[]; }; +#endif layout(binding = 6) buffer DstBuf { FLOAT_TYPE data_dst[]; }; #if !USE_SUBGROUP_ADD && !USE_SUBGROUP_CLUSTERED @@ -102,15 +114,26 @@ void main() { const uint iq3 = seq_id / rq3; const uint state_size = S_V * S_V; +#if USE_STATE_ROWS + // rows mode: state is a 2D cache view [D, n_rows]; row for this seq comes + // from the index buffer; row size (in floats) is H*state_size. + const uint row = uint(data_state_rows[seq_id]); + const uint state_in_base = row * H * state_size + head_id * state_size; +#else // input state holds s0 only [S_v, S_v, H, n_seqs]: per-seq stride is H*D. const uint state_in_base = (seq_id * H + head_id) * state_size; +#endif // output state layout per slot: same per-(seq,head) offset as the single-slot case. const uint state_out_base = (seq_id * H + head_id) * state_size; const uint state_size_per_snap = state_size * H * n_seqs; FLOAT_TYPE s_shard[ROWS_PER_LANE]; [[unroll]] for (uint r = 0; r < ROWS_PER_LANE; r++) { +#if STATE_BF16 + s_shard[r] = FLOAT_TYPE(state_load(state_in_base + col * S_V + r * LANES_PER_COLUMN + lane)); +#else s_shard[r] = FLOAT_TYPE(data_state[state_in_base + col * S_V + r * LANES_PER_COLUMN + lane]); +#endif } // snapshot slot mapping: slot 0 = most recent state, slot s = s tokens back. diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_ptq1_0.comp b/ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_ptq1_0.comp new file mode 100644 index 000000000000..9658056fa39e --- /dev/null +++ b/ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_ptq1_0.comp @@ -0,0 +1,123 @@ +#version 450 +#extension GL_EXT_shader_explicit_arithmetic_types_int32 : require + +#include "mul_mat_vec_base.glsl" + +layout(local_size_x_id = 0, local_size_y = 1, local_size_z = 1) in; + +FLOAT_TYPE temp[NUM_COLS][NUM_ROWS]; + +// PTQ1_0 dedicated matvec, v2 (algorithm ported from PrismML CUDA PR #218). +// Key ideas vs v1: +// - NUM_ROWS=4 rows per workgroup: every B-vector fetch is shared by 4 rows, +// cutting activation traffic 4x (the v1 kernel re-fetched B per row). +// - SIMD-in-register trit decode: 4 weights per uvec4 lane via +// (v*3 & 0xFF) >> 8 on all lanes at once, branchless. +// - Integer partial sums folded per 8-element group; one FMA per group. +// Thread map (16 lanes per block, workgroup = 64 => it_size = 4 blocks): +// itid 0.. 9 : region 1, elements e0 = itid*8 +// itid 10..14 : region 2, elements 80 + (itid-10)*8 +// itid 15 : region 3, elements 120..127 (qh) +void calc_superblock(const uint a_offset, const uint b_offset, const uint itid, const uint i, const uint num_blocks_per_row, const uint first_row, const uint num_rows) { + uint e0; + uint mi; + if (itid < 10u) { + e0 = itid * 8u; + mi = e0 >> 4u; + } else if (itid < 15u) { + e0 = 80u + (itid - 10u) * 8u; + mi = itid - 10u; + } else { + e0 = 120u; + mi = 0u; + } + + // tail guard: this thread's block may be past the end of a short row; + // return before touching B or A (1-3 block rows are legal in PTQ1_0) + if (i >= num_blocks_per_row) { return; } + + const uint y_idx = i * QUANT_K + e0; + + // B vectors fetched ONCE for all 4 rows + vec4 bv[NUM_COLS][2]; + [[unroll]] for (uint j = 0; j < NUM_COLS; ++j) { + bv[j][0] = vec4(data_b_v4[(j*p.batch_stride_b + b_offset + y_idx) / 4]); + bv[j][1] = vec4(data_b_v4[(j*p.batch_stride_b + b_offset + y_idx) / 4 + 1]); + } + + // weights decoded once per (row, group) + vec4 wf[4][2]; + + [[unroll]] for (uint n = 0; n < 4; ++n) { + if (n >= num_rows) { break; } + const uint ib0 = a_offset + (first_row+n)*num_blocks_per_row; + if (i >= num_blocks_per_row) { continue; } + + const FLOAT_TYPE d = FLOAT_TYPE(data_a[ib0 + i].d); + + if (itid < 15u) { + const uint qbase = (itid < 10u) ? (e0 & 15u) : (16u + ((e0 - 80u) & 7u)); + const uint mm = (mi == 0u) ? 1u : (mi == 1u) ? 3u : (mi == 2u) ? 9u : (mi == 3u) ? 27u : 81u; + const uvec4 q0 = uvec4(uint(data_a[ib0 + i].qs[qbase ]), + uint(data_a[ib0 + i].qs[qbase + 1u]), + uint(data_a[ib0 + i].qs[qbase + 2u]), + uint(data_a[ib0 + i].qs[qbase + 3u])); + const uvec4 q1 = uvec4(uint(data_a[ib0 + i].qs[qbase + 4u]), + uint(data_a[ib0 + i].qs[qbase + 5u]), + uint(data_a[ib0 + i].qs[qbase + 6u]), + uint(data_a[ib0 + i].qs[qbase + 7u])); + // SIMD trit step on 4 lanes at once: t = ((v*m)&0xFF)*3>>8, w = t-1 + const uvec4 t0 = (((q0 * mm) & uvec4(0xFFu)) * 3u) >> 8u; + const uvec4 t1 = (((q1 * mm) & uvec4(0xFFu)) * 3u) >> 8u; + wf[n][0] = vec4(t0) - 1.0f; + wf[n][1] = vec4(t1) - 1.0f; + } else { + const uint b0 = uint(data_a[ib0 + i].qh[0]); + const uint b1 = uint(data_a[ib0 + i].qh[1]); + const uvec4 t0 = ((uvec4(b0, b1, b0*3u, b1*3u ) & uvec4(0xFFu)) * 3u) >> 8u; + const uvec4 t1 = ((uvec4(b0*9u, b1*9u, b0*27u, b1*27u) & uvec4(0xFFu)) * 3u) >> 8u; + wf[n][0] = vec4(t0) - 1.0f; + wf[n][1] = vec4(t1) - 1.0f; + } + + [[unroll]] for (uint j = 0; j < NUM_COLS; ++j) { + temp[j][n] += (dot(wf[n][0], bv[j][0]) + dot(wf[n][1], bv[j][1])) * d; + } + } +} + +void compute_outputs(const uint32_t first_row, const uint32_t num_rows) { + uint a_offset, b_offset, d_offset; + get_offsets(a_offset, b_offset, d_offset); + + const uint num_blocks_per_row = p.ncols / QUANT_K; + + const uint it_size = gl_WorkGroupSize.x/16; + const uint tid = gl_LocalInvocationID.x; + const uint itid = tid%16; // 0...15 + const uint ix = tid/16; + + [[unroll]] for (uint j = 0; j < NUM_COLS; ++j) { + [[unroll]] for (uint i = 0; i < NUM_ROWS; ++i) { + temp[j][i] = FLOAT_TYPE(0); + } + } + + for (uint i0 = 0; i0 < num_blocks_per_row; i0 += it_size) + calc_superblock(a_offset, b_offset, itid, i0 + ix, num_blocks_per_row, first_row, num_rows); + + reduce_result(temp, d_offset, first_row, num_rows, tid); +} + +void main() { + const uint first_row = NUM_ROWS * (gl_WorkGroupID.x + gl_NumWorkGroups.x * gl_WorkGroupID.z); + + if (first_row + NUM_ROWS <= p.stride_d) { + compute_outputs(first_row, NUM_ROWS); + } else { + if (first_row >= p.stride_d) { + return; + } + compute_outputs(first_row, p.stride_d - first_row); + } +} diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/mul_mm_funcs.glsl b/ggml/src/ggml-vulkan/vulkan-shaders/mul_mm_funcs.glsl index 3125a3fcfa93..1ebd65855e2e 100644 --- a/ggml/src/ggml-vulkan/vulkan-shaders/mul_mm_funcs.glsl +++ b/ggml/src/ggml-vulkan/vulkan-shaders/mul_mm_funcs.glsl @@ -161,11 +161,52 @@ void load_a_to_shmem(const uint pos_a, const uint row, const uint col, const uin const float d = float(data_a[ib].d); + // FADI-OPT: an 8-element group shares one exponent n and reads + // consecutive bytes, so one multiplier + vector decode replaces + // 16 ptq1_0_trit() calls. NOTE: convert to float BEFORE the -1 + // (uint 0-1 wraps to 4.29e9, not -1). + uint base; + uint n; + if (grp < 10u) { + base = e0 & 15u; + n = e0 >> 4u; + } else if (grp < 15u) { + base = 16u + ((e0 - 80u) & 7u); + n = (e0 - 80u) >> 3u; + } else { + base = 0u; + n = 0u; + } + const uint m = (n == 0u) ? 1u : (n == 1u) ? 3u : (n == 2u) ? 9u : (n == 3u) ? 27u : 81u; + const uint k_pair = row * LOAD_VEC_A / 2; - [[unroll]] for (uint l = 0; l < 4; ++l) { - store_a(col, k_pair + l, FLOAT_TYPEV2( - ptq1_0_trit(ib, 0u, e0 + 2u*l) * d, - ptq1_0_trit(ib, 0u, e0 + 2u*l + 1u) * d)); + if (grp < 15u) { + const uvec4 i0 = (((uvec4(uint(data_a[ib].qs[base ]), + uint(data_a[ib].qs[base + 1u]), + uint(data_a[ib].qs[base + 2u]), + uint(data_a[ib].qs[base + 3u])) * m) & 0xFFu) * 3u) >> 8u; + const uvec4 i1 = (((uvec4(uint(data_a[ib].qs[base + 4u]), + uint(data_a[ib].qs[base + 5u]), + uint(data_a[ib].qs[base + 6u]), + uint(data_a[ib].qs[base + 7u])) * m) & 0xFFu) * 3u) >> 8u; + const vec4 f0 = (vec4(i0) - 1.0f) * d; + const vec4 f1 = (vec4(i1) - 1.0f) * d; + store_a(col, k_pair, FLOAT_TYPEV2(f0.x, f0.y)); + store_a(col, k_pair + 1, FLOAT_TYPEV2(f0.z, f0.w)); + store_a(col, k_pair + 2, FLOAT_TYPEV2(f1.x, f1.y)); + store_a(col, k_pair + 3, FLOAT_TYPEV2(f1.z, f1.w)); + } else { + // region 3: elements 120..127 alternate qh[0],qh[1] with paired exponents + const uint b0 = uint(data_a[ib].qh[0]); + const uint b1 = uint(data_a[ib].qh[1]); + const uvec4 j0 = (((uvec4(b0, b1, b0 * 3u, b1 * 3u )) & 0xFFu) * 3u) >> 8u; + const uvec4 j1 = (((uvec4(b0 * 9u, b1 * 9u, b0 * 27u, b1 * 27u)) & 0xFFu) * 3u) >> 8u; + const vec4 r0 = (vec4(j0) - 1.0f) * d; + const vec4 r1 = (vec4(j1) - 1.0f) * d; + store_a(col, k_pair, FLOAT_TYPEV2(r0.x, r0.y)); + store_a(col, k_pair + 1, FLOAT_TYPEV2(r0.z, r0.w)); + store_a(col, k_pair + 2, FLOAT_TYPEV2(r1.x, r1.y)); + store_a(col, k_pair + 3, FLOAT_TYPEV2(r1.z, r1.w)); } #elif defined(DATA_A_Q1_0) const uint idx = pos_a + col * p.stride_a / LOAD_VEC_A + row; diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/ptq1_0.glsl b/ggml/src/ggml-vulkan/vulkan-shaders/ptq1_0.glsl index 7eedd832b14f..0b5589e81148 100644 --- a/ggml/src/ggml-vulkan/vulkan-shaders/ptq1_0.glsl +++ b/ggml/src/ggml-vulkan/vulkan-shaders/ptq1_0.glsl @@ -12,6 +12,10 @@ // elements t*16+j, then an 8-byte chunk carrying 80 + t*8 + (j-16), then qh at four // trits per byte carrying 120 + t*2 + h. Trits come out by the base-3 remainder // recurrence t = (v*3)>>8, v = (v*3)&0xFF. + +// FADI-OPT: every region derives n <= 4, so five powers replace the serial recurrence loop. +const uint POW3_MOD256[5] = uint[5](1u, 3u, 9u, 27u, 81u); + float ptq1_0_trit(uint ib, uint a_offset, uint e) { uint b; uint n; @@ -28,10 +32,7 @@ float ptq1_0_trit(uint ib, uint a_offset, uint e) { n = t >> 1u; } - uint v = b; - for (uint i = 0u; i < n; ++i) { - v = (v * 3u) & 0xFFu; - } + const uint v = (b * POW3_MOD256[n]) & 0xFFu; return float(int((v * 3u) >> 8u) - 1); } diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp b/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp index 83af8af068fe..5de5271ae4f1 100644 --- a/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp +++ b/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp @@ -742,7 +742,7 @@ void process_shaders() { for (const auto& tname : type_names) { // mul mat vec std::string data_a_key = "DATA_A_" + to_uppercase(tname); - std::string shader = (string_ends_with(tname, "_k") || string_starts_with(tname, "iq1_") || string_starts_with(tname, "iq2_") || string_starts_with(tname, "iq3_") || tname == "tq2_0") ? "mul_mat_vec_" + tname + ".comp" : "mul_mat_vec.comp"; + std::string shader = (string_ends_with(tname, "_k") || string_starts_with(tname, "iq1_") || string_starts_with(tname, "iq2_") || string_starts_with(tname, "iq3_") || tname == "tq2_0" || tname == "ptq1_0") ? "mul_mat_vec_" + tname + ".comp" : "mul_mat_vec.comp"; string_to_spv("mul_mat_vec_" + tname + "_f32_f32", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}})); string_to_spv("mul_mat_vec_" + tname + "_f16_f32", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float16_t"}, {"B_TYPEV2", "f16vec2"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}})); @@ -902,6 +902,7 @@ void process_shaders() { string_to_spv("repeat_i16", "repeat.comp", {{"A_TYPE", "int16_t"}, {"D_TYPE", "int16_t"}}); string_to_spv("scale_f32", "scale.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}}); + string_to_spv("scale_bf16", "scale.comp", {{"A_TYPE", "uint16_t"}, {"D_TYPE", "uint16_t"}, {"FLOAT_TYPE", "float"}, {"SCALE_BF16", "1"}}); string_to_spv("pad_f32", "pad.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}}); string_to_spv("pad_reflect_1d_f32", "pad_reflect_1d.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}}); @@ -1082,6 +1083,15 @@ void process_shaders() { string_to_spv("gated_delta_net_f32", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "1"}})); string_to_spv("gated_delta_net_f32_nocluster", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "0"}})); string_to_spv("gated_delta_net_f32_shmem", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "0"}, {"USE_SUBGROUP_CLUSTERED", "0"}})); + string_to_spv("gated_delta_net_rows_f32", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "1"}, {"USE_STATE_ROWS", "1"}})); + string_to_spv("gated_delta_net_rows_f32_nocluster", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "0"}, {"USE_STATE_ROWS", "1"}})); + string_to_spv("gated_delta_net_rows_f32_shmem", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "0"}, {"USE_SUBGROUP_CLUSTERED", "0"}, {"USE_STATE_ROWS", "1"}})); + string_to_spv("gated_delta_net_rows_bf16state_f32", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "1"}, {"USE_STATE_ROWS", "1"}, {"STATE_BF16", "1"}})); + string_to_spv("gated_delta_net_rows_bf16state_rawgates_f32", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "1"}, {"USE_STATE_ROWS", "1"}, {"STATE_BF16", "1"}, {"RAW_GATES", "1"}})); + string_to_spv("gated_delta_net_rows_bf16state_f32_nocluster", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "0"}, {"USE_STATE_ROWS", "1"}, {"STATE_BF16", "1"}})); + string_to_spv("gated_delta_net_rows_bf16state_rawgates_f32_nocluster", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "0"}, {"USE_STATE_ROWS", "1"}, {"STATE_BF16", "1"}, {"RAW_GATES", "1"}})); + string_to_spv("gated_delta_net_rows_bf16state_f32_shmem", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "0"}, {"USE_SUBGROUP_CLUSTERED", "0"}, {"USE_STATE_ROWS", "1"}, {"STATE_BF16", "1"}})); + string_to_spv("gated_delta_net_rows_bf16state_rawgates_f32_shmem", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "0"}, {"USE_SUBGROUP_CLUSTERED", "0"}, {"USE_STATE_ROWS", "1"}, {"STATE_BF16", "1"}, {"RAW_GATES", "1"}})); string_to_spv("opt_step_adamw_f32", "opt_step_adamw.comp", merge_maps(base_dict, {{"A_TYPE", "float"}})); string_to_spv("opt_step_sgd_f32", "opt_step_sgd.comp", merge_maps(base_dict, {{"A_TYPE", "float"}})); diff --git a/ggml/src/ggml.c b/ggml/src/ggml.c index fa2a9c44f5cf..5af5f2cb39a5 100644 --- a/ggml/src/ggml.c +++ b/ggml/src/ggml.c @@ -6387,7 +6387,7 @@ struct ggml_tensor * ggml_gated_delta_net_rows( GGML_ASSERT(v->type == GGML_TYPE_F32); GGML_ASSERT(g->type == GGML_TYPE_F32); GGML_ASSERT(beta->type == GGML_TYPE_F32); - GGML_ASSERT(states->type == GGML_TYPE_F32); + GGML_ASSERT(states->type == GGML_TYPE_F32 || states->type == GGML_TYPE_BF16); GGML_ASSERT(rows->type == GGML_TYPE_I32); const int64_t S_v = v->ne[0]; diff --git a/src/llama-model.cpp b/src/llama-model.cpp index f30522c966d8..54c9e495e175 100644 --- a/src/llama-model.cpp +++ b/src/llama-model.cpp @@ -2771,6 +2771,10 @@ llama_memory_i * llama_model::create_memory(const llama_memory_params & params, /* filter_attn */ std::move(filter_attn), /* filter_recr */ std::move(filter_recr)); } else { + // bf16 SSM state pools: recurrence is bandwidth-bound on these; + // bf16 halves the traffic. Opt-in via env until kernels are audited. + const bool bf16_ssm_state = getenv("LLAMA_SSM_BF16_STATE") != nullptr; + const bool bf16_ssm_conv = getenv("LLAMA_SSM_BF16_CONV") != nullptr; res = new llama_memory_hybrid( /* model */ *this, /* attn_type_k */ params.type_k, @@ -2780,8 +2784,8 @@ llama_memory_i * llama_model::create_memory(const llama_memory_params & params, /* attn_n_pad */ 1, /* attn_n_swa */ hparams.n_swa, /* attn_swa_type */ hparams.swa_type, - /* recurrent_type_k */ GGML_TYPE_F32, - /* recurrent_type_v */ GGML_TYPE_F32, + /* recurrent_type_k */ bf16_ssm_conv ? GGML_TYPE_BF16 : GGML_TYPE_F32, + /* recurrent_type_v */ bf16_ssm_state ? GGML_TYPE_BF16 : GGML_TYPE_F32, /* recurrent_kv_size */ std::max((uint32_t) 1, cparams.n_seq_max), /* n_seq_max */ cparams.n_seq_max, /* n_rs_seq */ cparams.n_rs_seq, diff --git a/src/models/qwen35.cpp b/src/models/qwen35.cpp index 8e0944bc52ef..7f0d8d331720 100644 --- a/src/models/qwen35.cpp +++ b/src/models/qwen35.cpp @@ -1,5 +1,6 @@ #include "models.h" #include "llama-memory-recurrent.h" +#include "llama-impl.h" // llama_mul_mat_hadamard void llama_model_qwen35::load_arch_hparams(llama_model_loader & ml) { ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps); @@ -156,7 +157,7 @@ llama_model_qwen35::graph::graph(const llama_model & model, const llm_graph_para // integrated GPUs (e.g. unified-memory CUDA devices) report IGPU, not GPU const bool is_gpu = ggml_backend_dev_type(ldev.dev) == GGML_BACKEND_DEVICE_TYPE_GPU || ggml_backend_dev_type(ldev.dev) == GGML_BACKEND_DEVICE_TYPE_IGPU; - if (is_gpu && strcmp(reg_name, "MTL") != 0) { + if (is_gpu && strcmp(reg_name, "MTL") != 0 && strcmp(reg_name, "Vulkan") != 0) { gdn_state_rows_dev_ok = false; } if (strcmp(reg_name, "MTL") != 0 && strcmp(reg_name, "CUDA") != 0 && @@ -180,6 +181,7 @@ llama_model_qwen35::graph::graph(const llama_model & model, const llm_graph_para ggml_tensor * inp_pos = build_inp_pos(); ggml_tensor * inp_out_ids = build_inp_out_ids(); + // MTP/NextN layers are loaded as extra decoder blocks but not executed in the main pass. for (int il = 0; il < n_layer; ++il) { res->t_layer_inp[il] = inpL; @@ -229,6 +231,7 @@ llama_model_qwen35::graph::graph(const llama_model & model, const llm_graph_para // Input for next layer inpL = cur; + } cur = inpL; @@ -595,6 +598,20 @@ llama_model_qwen35::graph_mtp::graph_mtp(const llama_model & model, const llm_gr ggml_tensor * tok_embd_w = layer.nextn.embed_tokens ? layer.nextn.embed_tokens : model.tok_embd; tok_embd = ggml_get_rows(ctx0, tok_embd_w, inp->tokens); + + // a Hadamard-latent embedding table stores rotated rows; restore the + // primal basis right after the lookup: h = s * (H z). Mirrors + // build_inp_embd(); without this llama_verify_hadamard_graph rejects + // the MTP context on PrismML ternary models. + if (hadamard_inverses) { + const auto it = hadamard_inverses->find(tok_embd_w); + if (it != hadamard_inverses->end()) { + tok_embd = llama_mul_mat_hadamard(ctx0, tok_embd, it->second.rot); + if (it->second.signs) { + tok_embd = ggml_mul(ctx0, tok_embd, it->second.signs); + } + } + } } else { tok_embd = inp->embd; } diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cpp index 81cdcfe3a0c8..bd2e5bccddf5 100644 --- a/vendor/cpp-httplib/httplib.cpp +++ b/vendor/cpp-httplib/httplib.cpp @@ -1517,8 +1517,9 @@ bool mmap::open(const char *path) { if (wpath.empty()) { return false; } hFile_ = - ::CreateFile2(wpath.c_str(), GENERIC_READ, - FILE_SHARE_READ | FILE_SHARE_WRITE, OPEN_EXISTING, NULL); + ::CreateFileW(wpath.c_str(), GENERIC_READ, + FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, + FILE_ATTRIBUTE_NORMAL, NULL); if (hFile_ == INVALID_HANDLE_VALUE) { return false; } diff --git a/vendor/cpp-httplib/httplib.h b/vendor/cpp-httplib/httplib.h index 6fc86c7c75bf..7ef2a68c45c5 100644 --- a/vendor/cpp-httplib/httplib.h +++ b/vendor/cpp-httplib/httplib.h @@ -13,8 +13,7 @@ #ifdef _WIN32 #if defined(_WIN32_WINNT) && _WIN32_WINNT < 0x0A00 -#error \ - "cpp-httplib doesn't support Windows 8 or lower. Please use Windows 10 or later." +#define _WIN32_WINNT 0x0A00 #endif #endif