Skip to content

Unable to launch Strata on RTX 2060 6GB VRAM + 32 GB RAM #916

Closed as not planned
Closed as not planned
Unable to launch Strata on RTX 2060 6GB VRAM + 32 GB RAM#916

Description

@lollo78
lollo78
opened on Oct 5, 2026

**Hi, I'm trying to launch Strata also on my laptop, but it fails.

Here the full log:**

`Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU)

=== Step 1: checking your PC ===
[ok] GPU: NVIDIA GeForce RTX 2060, 6.0 GB VRAM, compute capability 7.5, driver 617.14
[!] less than 12 GB of VRAM: Strata will run, but most experts stay on the CPU and it will be slow
[ok] RAM: 32 GB
[ok] CPU: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz (AVX2)

=== Step 2: your choices ===

  1. Qwen3.8-Flash-Next Qwen; GSQ-RCO quants by ISTA-DASLab - the original model

  2. Swift 1.5 UkisAI's fine-tune of Qwen3.8-Flash-Next - thinks much shorter (-63% thinking tokens, 1.8x sooner answers by its authors' numbers)

  3. Qwen3.8-Flash-Next Coder ISTA-DASLab's coding version - half the experts (code, tools, images kept): needs ~32 GB of RAM, faster; weaker outside coding

  4. Qwen3.8-Flash-Next (Unsloth) Unsloth's ~4-bit quantizations - UD-IQ4_XS: a 94 GB download; with less than ~80 GB of RAM part of its experts are read from the SSD (UD-Q4_K_XL, 111 GB: experimental)
    Which model? [1]: 3
    [ok] model: Qwen3.8-Flash-Next Coder

  5. IQ1_M the Coder's only size: half the experts, stored like IQ3_S (3.5 bits); download 58 GB, uses ~23 GB of RAM <- fits in the low-RAM mode (the GPU holds ~4%, the rest from the SSD)
    Which size? [1]:
    [ok] size: IQ1_M

Context length = how much text the model can see at once (your chat, files, tool output).
Longer needs more VRAM for it, so fewer experts fit on the GPU:

  1. 8K tokens
  2. 32K tokens (recommended for your GPU)
  3. 64K tokens
  4. 128K tokens
  5. 256K tokens
  6. 384K tokens (experimental: setup adds rope scaling)
  7. 512K tokens (experimental: setup adds rope scaling)
    Context? [2]: 2
    [ok] context: 32768 tokens

KV cache precision (the model's memory of the conversation):

  1. 8-bit (recommended: what every published number was measured with)
  2. 4-bit half the memory (about 4% faster at 128K), but measurably less precise on long
    documents; long-context lookups (needle tests) still pass
    KV cache? [1]: 2
    [ok] KV cache: 4-bit (Hadamard-rotated)

Images: the model can also read pictures (screenshots, photos, scanned pages). This adds a 0.9 GB
download and keeps ~1.4 GB of VRAM free for the image encoder, so text is a few % slower.
Do you want images? [n]: n
[ok] images: off
[ok] low-RAM mode: IQ1_M's experts (23 GB) are read from the model folder through the OS file cache instead of a copy in RAM (32 GB); the GPU holds ~4% of them
[!] most of the experts are read from the SSD while it answers: expect it to be much slower than with enough RAM (a faster SSD and a smaller size help)

EXPERIMENTAL - speed projection: a small control vector applied while the model runs (layers 4-44).
It changes how the model answers: its package describes it as a refusal-direction projection (the
model declines far fewer requests). Off unless you choose it; when on, the web app can switch it off
per chat. Details: data/experimental-speed-projection/README.md
Turn on the experimental speed projection? [n]:
[ok] experimental speed projection: off

=== Step 3: Python packages ===
Installing numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow, psutil ...

D:\Strata-main.venv\Scripts\python.exe -m pip install --quiet --disable-pip-version-check numpy==2.5.3; python_version >= "3.12" numpy==2.4.6; python_version == "3.11" numpy==2.2.6; python_version < "3.11" jinja2==3.1.6 regex==2026.9.10 pyyaml==6.0.3 tqdm==4.70.1 requests==2.34.2 cmake==4.4.3 ninja==1.13.2 pillow==12.3.0 psutil==7.2.2 markupsafe==3.0.3 certifi==2026.7.22 charset-normalizer==3.5.1 idna==3.20 urllib3==2.8.0 colorama==0.4.6; sys_platform == "win32"
[ok] numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow, psutil installed

=== Step 4: the Strata engine ===
llama.cpp source: 0.04 / 0.04 GB (100%)
[ok] llama.cpp source downloaded
[ok] llama.cpp 3cf0325 (gguf-py, ggml, mtmd)
Downloading the ready-made Strata engine ...
Strata engine: 0.09 / 0.12 GB (75%)
[ok] Strata engine downloaded
[ok] ready-made engine 0.1.39 for sm_75, sm_86, sm_89, sm_120 (CUDA 13.0)
Installing NVIDIA CUDA libraries (cuBLAS, CUDA runtime; ~0.4 GB) ...

D:\Strata-main.venv\Scripts\python.exe -m pip install --quiet --disable-pip-version-check nvidia-cublas==13.0.2.14 nvidia-cuda-runtime==13.0.96
[ok] NVIDIA CUDA libraries (cuBLAS, CUDA runtime; ~0.4 GB) installed
[ok] engine: D:\Strata-main\engine\strata.exe

=== Step 5: downloading Qwen3.8-Flash-Next Coder IQ1_M ===
The model files go in D:\Strata-data\models\coder-IQ1_M
Files you already have: put them here with their original names (Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf, Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf), or use --gguf-dir .
Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf: 29.59 / 29.61 GB (100%)
[ok] Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf downloaded
Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf: 28.78 / 28.80 GB (100%)
[ok] Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf downloaded
[ok] model files present

=== Step 6: preparing the model for Strata ===

D:\Strata-main.venv\Scripts\python.exe D:\Strata-main\tools\iq_pack.py --gguf D:\Strata-data\models\coder-IQ1_M\Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf --out D:\Strata-data\packs\coder-iq1_m
model shards: Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf, Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
index.txt: 1079 tensors, 302 served natively, 0 in extra.bin, arena 1.37 GiB
tokenizer/: vocab 248320, merges 247587, pre qwen35, specials {'tokenizer.ggml.eos_token_id': 248046, 'tokenizer.ggml.padding_token_id': 248044, 'tokenizer.ggml.bos_token_id': 248044}
Writing the experts into one file for the low-RAM mode (one time, 23 GB) ...
D:\Strata-main.venv\Scripts\python.exe D:\Strata-main\tools\iq_pack.py --gguf D:\Strata-data\models\coder-IQ1_M\Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf --out D:\Strata-data\packs\coder-iq1_m --experts-bin
model shards: Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf, Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
index.txt: 1079 tensors, 302 served natively, 0 in extra.bin, arena 1.37 GiB
layer 0 IQ3_XXS /IQ4_NL blob 2176000 at 0.00 GiB
layer 8 IQ2_S /Q2_0 blob 1510400 at 3.75 GiB
layer 16 IQ3_XXS /IQ4_NL blob 2176000 at 7.17 GiB
layer 24 IQ2_S /IQ4_NL blob 1971200 at 11.07 GiB
layer 32 IQ2_S /IQ4_NL blob 1971200 at 15.16 GiB
layer 40 IQ3_S /IQ4_NL blob 2329600 at 19.04 GiB
experts.bin: 48 layers, 23.42 GiB
[ok] model prepared: D:\Strata-data\packs\coder-iq1_m
The MTP draft layer (speculative decoding, ~2x faster output) comes from the original Qwen checkpoint:
only its ~5 GB of MTP tensors are downloaded.
D:\Strata-main.venv\Scripts\python.exe D:\Strata-main\tools\mtp_fetch.py fetch --out D:\Strata-data\mtp

MTP block in the BF16 checkpoint

31 tensors, 5.214 GB, in 28 shards.

tensor dtype shape MB
mtp.fc_embedding.weight BF16 2560x2560 13.1
mtp.fc_hidden.weight BF16 2560x2560 13.1
mtp.hyper_connection_mixer.hc_norm.weight BF16 10240 0.0
mtp.hyper_connection_mixer.input_mix_weight_down.weight BF16 320x10240 6.6
mtp.hyper_connection_mixer.input_mix_weight_up.weight BF16 10240x320 6.6
mtp.layers.0.attn_hyper_connection.block_inject_weight.weight BF16 4x10240 0.1
mtp.layers.0.attn_hyper_connection.hc_norm.weight BF16 10240 0.0
mtp.layers.0.attn_hyper_connection.input_mix_weight_down.weight BF16 320x10240 6.6
mtp.layers.0.attn_hyper_connection.input_mix_weight_up.weight BF16 10240x320 6.6
mtp.layers.0.mlp.experts.down_proj BF16 512x2560x640 1677.7
mtp.layers.0.mlp.experts.gate_up_proj BF16 512x1280x2560 3355.4
mtp.layers.0.mlp.gate.weight BF16 512x2560 2.6
mtp.layers.0.mlp.shared_expert.down_proj.weight BF16 2560x640 3.3
mtp.layers.0.mlp.shared_expert.gate_proj.weight BF16 640x2560 3.3
mtp.layers.0.mlp.shared_expert.up_proj.weight BF16 640x2560 3.3
mtp.layers.0.mlp.shared_expert_gate.weight BF16 1x2560 0.0
mtp.layers.0.mlp_hyper_connection.block_inject_weight.weight BF16 4x10240 0.1
mtp.layers.0.mlp_hyper_connection.hc_norm.weight BF16 10240 0.0
mtp.layers.0.mlp_hyper_connection.input_mix_weight_down.weight BF16 320x10240 6.6
mtp.layers.0.mlp_hyper_connection.input_mix_weight_up.weight BF16 10240x320 6.6
mtp.layers.0.self_attn.indexer.index_qk_proj.weight BF16 640x2560 3.3
mtp.layers.0.self_attn.indexer.k_layernorm.weight BF16 128 0.0
mtp.layers.0.self_attn.indexer.q_layernorm.weight BF16 128 0.0
mtp.layers.0.self_attn.k_norm.weight BF16 256 0.0
mtp.layers.0.self_attn.k_proj.weight BF16 512x2560 2.6
mtp.layers.0.self_attn.o_proj.weight BF16 2560x6144 31.5
mtp.layers.0.self_attn.q_norm.weight BF16 256 0.0
mtp.layers.0.self_attn.q_proj.weight BF16 12288x2560 62.9
mtp.layers.0.self_attn.v_proj.weight BF16 512x2560 2.6
mtp.pre_fc_norm_embedding.weight BF16 2560 0.0
mtp.pre_fc_norm_hidden.weight BF16 10240 0.0

mtp.fc_embedding.weight 100%
mtp.fc_hidden.weight 100%
mtp.hyper_connection_mixer.hc_norm.weight 100%
mtp.hyper_connection_mixer.input_mix_weight_down.weight 100%
mtp.hyper_connection_mixer.input_mix_weight_up.weight 100%
mtp.layers.0.attn_hyper_connection.block_inject_weight.weight 100%
mtp.layers.0.attn_hyper_connection.hc_norm.weight 100%
mtp.layers.0.attn_hyper_connection.input_mix_weight_down.weight 100%
mtp.layers.0.attn_hyper_connection.input_mix_weight_up.weight 100%
mtp.layers.0.mlp.experts.down_proj 4%
mtp.layers.0.mlp.experts.down_proj 8%
mtp.layers.0.mlp.experts.down_proj 12%
mtp.layers.0.mlp.experts.down_proj 16%
mtp.layers.0.mlp.experts.down_proj 20%
mtp.layers.0.mlp.experts.down_proj 24%
mtp.layers.0.mlp.experts.down_proj 28%
mtp.layers.0.mlp.experts.down_proj 32%
mtp.layers.0.mlp.experts.down_proj 36%
mtp.layers.0.mlp.experts.down_proj 40%
mtp.layers.0.mlp.experts.down_proj 44%
mtp.layers.0.mlp.experts.down_proj 48%
mtp.layers.0.mlp.experts.down_proj 52%
mtp.layers.0.mlp.experts.down_proj 56%
mtp.layers.0.mlp.experts.down_proj 60%
mtp.layers.0.mlp.experts.down_proj 64%
mtp.layers.0.mlp.experts.down_proj 68%
mtp.layers.0.mlp.experts.down_proj 72%
mtp.layers.0.mlp.experts.down_proj 76%
mtp.layers.0.mlp.experts.down_proj 80%
mtp.layers.0.mlp.experts.down_proj 84%
mtp.layers.0.mlp.experts.down_proj 88%
mtp.layers.0.mlp.experts.down_proj 92%
mtp.layers.0.mlp.experts.down_proj 96%
mtp.layers.0.mlp.experts.down_proj 100%
mtp.layers.0.mlp.experts.gate_up_proj 2%
mtp.layers.0.mlp.experts.gate_up_proj 4%
mtp.layers.0.mlp.experts.gate_up_proj 6%
mtp.layers.0.mlp.experts.gate_up_proj 8%
mtp.layers.0.mlp.experts.gate_up_proj 10%
mtp.layers.0.mlp.experts.gate_up_proj 12%
mtp.layers.0.mlp.experts.gate_up_proj 14%
mtp.layers.0.mlp.experts.gate_up_proj 16%
mtp.layers.0.mlp.experts.gate_up_proj 18%
mtp.layers.0.mlp.experts.gate_up_proj 20%
mtp.layers.0.mlp.experts.gate_up_proj 22%
mtp.layers.0.mlp.experts.gate_up_proj 24%
mtp.layers.0.mlp.experts.gate_up_proj 26%
mtp.layers.0.mlp.experts.gate_up_proj 28%
mtp.layers.0.mlp.experts.gate_up_proj 30%
mtp.layers.0.mlp.experts.gate_up_proj 32%
mtp.layers.0.mlp.experts.gate_up_proj 34%
mtp.layers.0.mlp.experts.gate_up_proj 36%
mtp.layers.0.mlp.experts.gate_up_proj 38%
mtp.layers.0.mlp.experts.gate_up_proj 40%
mtp.layers.0.mlp.experts.gate_up_proj 42%
mtp.layers.0.mlp.experts.gate_up_proj 44%
mtp.layers.0.mlp.experts.gate_up_proj 46%
mtp.layers.0.mlp.experts.gate_up_proj 48%
mtp.layers.0.mlp.experts.gate_up_proj 50%
mtp.layers.0.mlp.experts.gate_up_proj 52%
mtp.layers.0.mlp.experts.gate_up_proj 54%
mtp.layers.0.mlp.experts.gate_up_proj 56%
mtp.layers.0.mlp.experts.gate_up_proj 58%
mtp.layers.0.mlp.experts.gate_up_proj 60%
mtp.layers.0.mlp.experts.gate_up_proj 62%
mtp.layers.0.mlp.experts.gate_up_proj 64%
mtp.layers.0.mlp.experts.gate_up_proj 66%
mtp.layers.0.mlp.experts.gate_up_proj 68%
mtp.layers.0.mlp.experts.gate_up_proj 70%
mtp.layers.0.mlp.experts.gate_up_proj 72%
mtp.layers.0.mlp.experts.gate_up_proj 74%
mtp.layers.0.mlp.experts.gate_up_proj 76%
mtp.layers.0.mlp.experts.gate_up_proj 78%
mtp.layers.0.mlp.experts.gate_up_proj 80%
mtp.layers.0.mlp.experts.gate_up_proj 82%
mtp.layers.0.mlp.experts.gate_up_proj 84%
mtp.layers.0.mlp.experts.gate_up_proj 86%
mtp.layers.0.mlp.experts.gate_up_proj 88%
mtp.layers.0.mlp.experts.gate_up_proj 90%
mtp.layers.0.mlp.experts.gate_up_proj 92%
mtp.layers.0.mlp.experts.gate_up_proj 94%
mtp.layers.0.mlp.experts.gate_up_proj 96%
mtp.layers.0.mlp.experts.gate_up_proj 98%
mtp.layers.0.mlp.experts.gate_up_proj 100%
mtp.layers.0.mlp.gate.weight 100%
mtp.layers.0.mlp.shared_expert.down_proj.weight 100%
mtp.layers.0.mlp.shared_expert.gate_proj.weight 100%
mtp.layers.0.mlp.shared_expert.up_proj.weight 100%
mtp.layers.0.mlp.shared_expert_gate.weight 100%
mtp.layers.0.mlp_hyper_connection.block_inject_weight.weight 100%
mtp.layers.0.mlp_hyper_connection.hc_norm.weight 100%
mtp.layers.0.mlp_hyper_connection.input_mix_weight_down.weight 100%
mtp.layers.0.mlp_hyper_connection.input_mix_weight_up.weight 100%
mtp.layers.0.self_attn.indexer.index_qk_proj.weight 100%
mtp.layers.0.self_attn.indexer.k_layernorm.weight 100%
mtp.layers.0.self_attn.indexer.q_layernorm.weight 100%
mtp.layers.0.self_attn.k_norm.weight 100%
mtp.layers.0.self_attn.k_proj.weight 100%
mtp.layers.0.self_attn.o_proj.weight 100%
mtp.layers.0.self_attn.q_norm.weight 100%
mtp.layers.0.self_attn.q_proj.weight 100%
mtp.layers.0.self_attn.v_proj.weight 100%
mtp.pre_fc_norm_embedding.weight 100%
mtp.pre_fc_norm_hidden.weight 100%

D:\Strata-main.venv\Scripts\python.exe D:\Strata-main\tools\mtp_pack.py --src D:\Strata-data\mtp --experts q2_0 --out D:\Strata-data\mtp\mtp-q2_0.gguf
mtp.fc_embedding.weight: BF16 13.1 MB
mtp.fc_hidden.weight: BF16 13.1 MB
mtp.hyper_connection_mixer.hc_norm.weight: F32 0.0 MB
mtp.hyper_connection_mixer.input_mix_weight_down.weight: BF16 6.6 MB
mtp.hyper_connection_mixer.input_mix_weight_up.weight: BF16 6.6 MB
mtp.layers.0.attn_hyper_connection.block_inject_weight.weight: BF16 0.1 MB
mtp.layers.0.attn_hyper_connection.hc_norm.weight: F32 0.0 MB
mtp.layers.0.attn_hyper_connection.input_mix_weight_down.weight: BF16 6.6 MB
mtp.layers.0.attn_hyper_connection.input_mix_weight_up.weight: BF16 6.6 MB
mtp.layers.0.mlp.experts.down_proj: q2_0 235.9 MB
mtp.layers.0.mlp.experts.gate_up_proj: q2_0 471.9 MB
mtp.layers.0.mlp.gate.weight: BF16 2.6 MB
mtp.layers.0.mlp.shared_expert.down_proj.weight: BF16 3.3 MB
mtp.layers.0.mlp.shared_expert.gate_proj.weight: BF16 3.3 MB
mtp.layers.0.mlp.shared_expert.up_proj.weight: BF16 3.3 MB
mtp.layers.0.mlp.shared_expert_gate.weight: BF16 0.0 MB
mtp.layers.0.mlp_hyper_connection.block_inject_weight.weight: BF16 0.1 MB
mtp.layers.0.mlp_hyper_connection.hc_norm.weight: F32 0.0 MB
mtp.layers.0.mlp_hyper_connection.input_mix_weight_down.weight: BF16 6.6 MB
mtp.layers.0.mlp_hyper_connection.input_mix_weight_up.weight: BF16 6.6 MB
mtp.layers.0.self_attn.indexer.index_qk_proj.weight: BF16 3.3 MB
mtp.layers.0.self_attn.indexer.k_layernorm.weight: F32 0.0 MB
mtp.layers.0.self_attn.indexer.q_layernorm.weight: F32 0.0 MB
mtp.layers.0.self_attn.k_norm.weight: F32 0.0 MB
mtp.layers.0.self_attn.k_proj.weight: BF16 2.6 MB
mtp.layers.0.self_attn.o_proj.weight: BF16 31.5 MB
mtp.layers.0.self_attn.q_norm.weight: F32 0.0 MB
mtp.layers.0.self_attn.q_proj.weight: BF16 62.9 MB
mtp.layers.0.self_attn.v_proj.weight: BF16 2.6 MB
mtp.pre_fc_norm_embedding.weight: F32 0.0 MB
mtp.pre_fc_norm_hidden.weight: F32 0.0 MB
wrote D:\Strata-data\mtp\mtp-q2_0.gguf: 0.889 GB of tensor data
D:\Strata-main.venv\Scripts\python.exe D:\Strata-main\tools\mtp_rt.py --gguf D:\Strata-data\mtp\mtp-q2_0.gguf --out D:\Strata-data\mtp\rt
experts.bin 707788800 B, dense.bin 116099072 B, 29 tensors -> D:\Strata-data\mtp\rt
fc_embedding.weight q8_0 2560 2560 0 6963200
fc_hidden.weight q8_0 2560 2560 6963200 6963200
hyper_connection_mixer.hc_norm.weight f32 1 10240 13926400 40960
hyper_connection_mixer.input_mix_weight_down.weight bf16 320 10240 13967360 6553600
hyper_connection_mixer.input_mix_weight_up.weight bf16 10240 320 20520960 6553600
attn_hyper_connection.block_inject_weight.weight bf16 4 10240 27074560 81920
attn_hyper_connection.hc_norm.weight f32 1 10240 27156480 40960
attn_hyper_connection.input_mix_weight_down.weight bf16 320 10240 27197440 6553600
attn_hyper_connection.input_mix_weight_up.weight bf16 10240 320 33751040 6553600
mlp.gate.weight bf16 512 2560 40304640 2621440
mlp.shared_expert.down_proj.weight q8_0 2560 640 42926080 1740800
mlp.shared_expert.gate_proj.weight q8_0 640 2560 44666880 1740800
mlp.shared_expert.up_proj.weight q8_0 640 2560 46407680 1740800
mlp.shared_expert_gate.weight bf16 1 2560 48148480 5120
mlp_hyper_connection.block_inject_weight.weight bf16 4 10240 48153600 81920
mlp_hyper_connection.hc_norm.weight f32 1 10240 48235520 40960
mlp_hyper_connection.input_mix_weight_down.weight bf16 320 10240 48276480 6553600
mlp_hyper_connection.input_mix_weight_up.weight bf16 10240 320 54830080 6553600
self_attn.indexer.index_qk_proj.weight q8_0 640 2560 61383680 1740800
self_attn.indexer.k_layernorm.weight f32 1 128 63124480 512
self_attn.indexer.q_layernorm.weight f32 1 128 63124992 512
self_attn.k_norm.weight f32 1 256 63125504 1024
self_attn.k_proj.weight q8_0 512 2560 63126528 1392640
self_attn.o_proj.weight q8_0 2560 6144 64519168 16711680
self_attn.q_norm.weight f32 1 256 81230848 1024
self_attn.q_proj.weight q8_0 12288 2560 81231872 33423360
self_attn.v_proj.weight q8_0 512 2560 114655232 1392640
pre_fc_norm_embedding.weight f32 1 2560 116047872 10240
pre_fc_norm_hidden.weight f32 1 10240 116058112 40960
[ok] MTP draft layer: D:\Strata-data\mtp\rt
Tip for a 6 GB card: the draft layer's default token subset (with Chinese, Japanese and Korean) needs up to ~348 MiB of VRAM.
For English and code answers, START-HERE.bat --draft-vocab en needs up to ~133 MiB (cyrillic: ~193) and leaves the rest to the expert cache - and it is the
fix when the start stops with "the draft head does not fit". The model keeps the choice.

=== Step 7: writing the start script ===
If the start stops with "no VRAM is left for the expert cache" (the engine's log says how much is short):
run START-HERE.bat --setup again with an 8K context (a smaller KV cache) and --draft-vocab en (a draft head of ~133 MiB instead of ~348), or close other programs that use the GPU.
Tune Strata for this PC now? It measures a few engine settings (about 5-10 minutes; the PC is busy meanwhile; later: START-HERE --calibrate) [y]: y

Tuning Strata for this PC: the output speed is measured with a few engine settings (the PCIe share, the
draft depth, the CPU threads). It takes about 5-10 minutes; the PC is busy meanwhile.
Loading the model for the measurements ...
[strata] starting the engine: reading the model's weights ...
[strata] mapping the experts from the model files (about 30 GB, --mmap-experts): they are not loaded into RAM - the OS file cache reads them as the GPU's expert cache fills and as requests need them.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] almost ready ...
[!] the tuning did not finish (the engine exited before it was ready (see D:\Strata-main\strata-coder-iq1_m.log)
the engine log's last lines:
strata generate: PCIe probe: 12.7 GB/s host->device (best of 12.7 12.7 12.7 12.7) -> pcie_frac 0.35 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: D:\Strata-data\packs\coder-iq1_m experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB)
strata generate: 1406 MiB of weights loaded from D:\Strata-data\packs\coder-iq1_m (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: profile D:\Strata-main\data\expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots
strata generate: PLE on, table 320001536 rows of D:\Strata-data\models\coder-IQ1_M\Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
strata mtp: draft layer loaded, 824 MiB of VRAM (experts 675, dense 111), files read in 0.52 s (1525 MiB/s)
strata generate: experimental native Q5_K head, 521472000 bytes
strata generate: GPU 0: NVIDIA GeForce RTX 2060, compute capability 7.5
strata generate: experts via mmap (--mmap-experts; the A/B arm of R2.1)
strata generate: expert cache auto: 0.00 GiB free, 700 MiB reserved (+218 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 241 slots (611 MiB), about 1131 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 393 MiB at 32768 tokens), --kv q4_0, the English draft subset (setup --draft-vocab en; the draft head takes 218 MiB now), images on the CPU, or close other programs that use the GPU
strata generate: pre-filled 0 of 0 slots from the profile; slot 0 verified
strata generate: 5 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.39)
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata serve: verify: needs the profile-filled VRAM expert tier (--expert-profile and --expert-cache); with --expert-cache auto, no VRAM was left for it - the 'no VRAM is left for the expert cache' line above says how much is short and what makes room (a smaller --max-context, --kv q4_0, setup --draft-vocab en, images on the CPU)): the default settings stay
the engine said: strata serve: verify: needs the profile-filled VRAM expert tier (--expert-profile and --expert-cache); with --expert-cache auto, no VRAM was left for it - the 'no VRAM is left for the expert cache' line above says how much is short and what makes room (a smaller --max-context, --kv q4_0, setup --draft-vocab en, images on the CPU)
[ok] start script: run-coder-iq1_m.bat

All set.
API (OpenAI): http://127.0.0.1:8080/v1 (any API key; model name: anything)
API (Anthropic): http://127.0.0.1:8080/v1/messages
Next time: just run START-HERE.bat (or run-coder-iq1_m.bat) - it starts right away
Several requests at once: left at one at a time - parallel N reduces waiting for several users but costs about 10-25% speed per request on this card (docs/BATCHING.md).
Tuning: FAILED (the reason is above): the default settings stay - START-HERE.bat --calibrate tries again
[ok] GPU: GPU 0 (NVIDIA GeForce RTX 2060, 6 GB)

Starting qwen3.8-flash-next-coder-iq1_m: it maps about 30 GB of experts from the model files (the OS file cache reads them).
While it does, YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES (longer the first time after a
restart). That is normal: please wait and don't close this window - the browser opens when it is ready.
Later, closing this window stops the model.

Settings (strata-coder-iq1_m.json): --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
--max-context 32768 --kv q4_0 --mmap-experts; server 127.0.0.1:8080, gpu 0
loading the model (the first start takes a minute or two) ...
[strata] starting the engine: reading the model's weights ...
[strata] mapping the experts from the model files (about 30 GB, --mmap-experts): they are not loaded into RAM - the OS file cache reads them as the GPU's expert cache fills and as requests need them.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] almost ready ...
Traceback (most recent call last):
File "D:\Strata-main\serve\server.py", line 4145, in
sys.exit(main())
^^^^^^
File "D:\Strata-main\serve\server.py", line 4001, in main
engine = StrataEngine(exe, engine_args(cfg) + (effort_end or []), cwd=cfg.get("cwd"), log=cfg.get("log"),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Strata-main\serve\server.py", line 478, in init
raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "") +
RuntimeError: the engine exited before it was ready (see D:\Strata-main\strata-coder-iq1_m.log)
the engine log's last lines:
strata generate: PCIe probe: 12.6 GB/s host->device (best of 12.6 12.5 12.5 12.5) -> pcie_frac 0.35 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: D:\Strata-data\packs\coder-iq1_m experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB)
strata generate: 1406 MiB of weights loaded from D:\Strata-data\packs\coder-iq1_m (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: profile D:\Strata-main\data\expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots
strata generate: PLE on, table 320001536 rows of D:\Strata-data\models\coder-IQ1_M\Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
strata mtp: draft layer loaded, 824 MiB of VRAM (experts 675, dense 111), files read in 0.40 s (1968 MiB/s)
strata generate: experimental native Q5_K head, 521472000 bytes
strata generate: GPU 0: NVIDIA GeForce RTX 2060, compute capability 7.5
strata generate: experts via mmap (--mmap-experts; the A/B arm of R2.1)
strata generate: expert cache auto: 0.00 GiB free, 700 MiB reserved (+218 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 241 slots (611 MiB), about 1131 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 393 MiB at 32768 tokens), --kv q4_0, the English draft subset (setup --draft-vocab en; the draft head takes 218 MiB now), images on the CPU, or close other programs that use the GPU
strata generate: pre-filled 0 of 0 slots from the profile; slot 0 verified
strata generate: 5 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.39)
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata serve: verify: needs the profile-filled VRAM expert tier (--expert-profile and --expert-cache); with --expert-cache auto, no VRAM was left for it - the 'no VRAM is left for the expert cache' line above says how much is short and what makes room (a smaller --max-context, --kv q4_0, setup --draft-vocab en, images on the CPU)
Premere un tasto per continuare . . .`

I tried closing some apps (to free VRAM) and using the en vocab but nothing:

`D:\Strata-main>START-HERE.bat --draft-vocab en
Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU)
[ok] GPU: GPU 0 (NVIDIA GeForce RTX 2060, 6 GB)

Starting qwen3.8-flash-next-coder-iq1_m: it maps about 30 GB of experts from the model files (the OS file cache reads them).
While it does, YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES (longer the first time after a
restart). That is normal: please wait and don't close this window - the browser opens when it is ready.
Later, closing this window stops the model.

Settings (strata-coder-iq1_m.json): --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
--max-context 32768 --kv q4_0 --mmap-experts; server 127.0.0.1:8080, gpu 0, draft_vocab en
loading the model (the first start takes a minute or two) ...
[strata] starting the engine: reading the model's weights ...
[strata] mapping the experts from the model files (about 30 GB, --mmap-experts): they are not loaded into RAM - the OS file cache reads them as the GPU's expert cache fills and as requests need them.
YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES NOW - this is normal.
Please wait and don't close this window; the browser opens when it is ready.
[strata] almost ready ...
Traceback (most recent call last):
File "D:\Strata-main\serve\server.py", line 4145, in
sys.exit(main())
^^^^^^
File "D:\Strata-main\serve\server.py", line 4001, in main
engine = StrataEngine(exe, engine_args(cfg) + (effort_end or []), cwd=cfg.get("cwd"), log=cfg.get("log"),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Strata-main\serve\server.py", line 478, in init
raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "") +
RuntimeError: the engine exited before it was ready (see D:\Strata-main\strata-coder-iq1_m.log)
the engine log's last lines:
strata generate: PCIe probe: 12.7 GB/s host->device (best of 12.7 12.7 12.7 12.7) -> pcie_frac 0.35 (default 0.55)
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: D:\Strata-data\packs\coder-iq1_m experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB)
strata generate: 1406 MiB of weights loaded from D:\Strata-data\packs\coder-iq1_m (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: profile D:\Strata-main\data\expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots
strata generate: PLE on, table 320001536 rows of D:\Strata-data\models\coder-IQ1_M\Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
strata mtp: draft layer loaded, 824 MiB of VRAM (experts 675, dense 111), files read in 0.37 s (2131 MiB/s)
strata generate: experimental native Q5_K head, 521472000 bytes
strata generate: GPU 0: NVIDIA GeForce RTX 2060, compute capability 7.5
strata generate: experts via mmap (--mmap-experts; the A/B arm of R2.1)
strata generate: expert cache auto: 0.00 GiB free, 700 MiB reserved (+86 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 241 slots (611 MiB), about 999 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 393 MiB at 32768 tokens), --kv q4_0, the English draft subset (setup --draft-vocab en; the draft head takes 86 MiB now), images on the CPU, or close other programs that use the GPU
strata generate: pre-filled 0 of 0 slots from the profile; slot 0 verified
strata generate: 5 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.39)
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata serve: verify: needs the profile-filled VRAM expert tier (--expert-profile and --expert-cache); with --expert-cache auto, no VRAM was left for it - the 'no VRAM is left for the expert cache' line above says how much is short and what makes room (a smaller --max-context, --kv q4_0, setup --draft-vocab en, images on the CPU)
Premere un tasto per continuare . . .`

Any advise?
Thanks

Activity

  1. lollo78 commented on Oct 5, 2026

    @lollo78
    lollo78
    on Oct 5, 2026
    Author

    I tried also with 8K max contest, but nothing.

  2. grigoriitropin commented on Oct 6, 2026

    @grigoriitropin
    grigoriitropin
    on Oct 6, 2026

    i was able to run iq2_xs with 64k context on 6gb rtx 4050, are you using nvidia optimus or have you disabled the igpu? (you are out of vram)

  3. lollo78 commented on Oct 6, 2026

    @lollo78
    lollo78
    on Oct 6, 2026
    Author

    Hi, I can't use iq2_xs due to my RAM (only 32GB). For this reason I'm on coder-iq1_m.
    Seems that here are no issue with Nvidia Optimus. My system use igpu and leave dgpu totally free:

    Image

    I don't know what to do...

  4. grigoriitropin commented on Oct 6, 2026

    @grigoriitropin
    grigoriitropin
    on Oct 6, 2026

    probably wait until smaller models get better, iq1 quant can't really do any hard tasks, context is to small etc etc.

  5. grigoriitropin commented on Oct 7, 2026

    @grigoriitropin
    grigoriitropin
    on Oct 7, 2026

    actually there is something that works for me, was able to disable mtp, and using the vram i recieved, i was able to switch model to iq3_xss and expand context to 131k. performance dropped from 145 t/s on prefill and 25 t/s on decode to something about 120 t/s on prefill and 20 t/s on decode, but for better quantization and more context imo it worth it.

    having flag to disable mtp on systems with low amounts of vram would be usefull
    P.S. was able to extend context window to 180k

  6. sathya731 commented on Oct 7, 2026

    @sathya731
    sathya731
    on Oct 7, 2026

    @grigoriitropin I am also having 4050 6 GB but ram is just 24 GB.
    What which model are you running pruned coder iq3_XSS or vanilla qwen iq3_XSS?
    Do you think it will fit in 24 GB RAM?

  7. grigoriitropin commented on Oct 7, 2026

    @grigoriitropin
    grigoriitropin
    on Oct 7, 2026

    i have 64gb of ddr4 ram and model consumes 44gb. im running vanilla iq3_xss. 24gb... you can try coder version, i haven't tested it but in setup script it says it needs 23gb of ram, you can try i guess, but not sure about how it will perform. however can't say how much vram it needs

    but this is both under vram and ram requirements, so wouldn't count on it running well

  8. djismgaming commented on Oct 8, 2026

    @djismgaming
    djismgaming
    on Oct 8, 2026

    actually there is something that works for me, was able to disable mtp, and using the vram i recieved, i was able to switch model to iq3_xss and expand context to 131k. performance dropped from 145 t/s on prefill and 25 t/s on decode to something about 120 t/s on prefill and 20 t/s on decode, but for better quantization and more context imo it worth it.

    having flag to disable mtp on systems with low amounts of vram would be usefull P.S. was able to extend context window to 180k

    How did you disable the mtp?

  9. grigoriitropin commented on Oct 8, 2026

    @grigoriitropin
    grigoriitropin
    on Oct 8, 2026

    to disable mtp.... you have to change source code. i asked ai to do it. here is short recap:
    Currently,there is no official CLI flag to disable MTP in server mode becausestrata
    servehard-requires--mtp

    ` and assumes speculative drafting is active.

    To completely disable MTP (saving ~800–900 MB of VRAM on 6 GB cards), you only need a

    small patch in src/program/generate.cpp and a quick recompile:

    ### 1. Patch `src/program/generate.cpp`                                               
                                                                                      
* **Trigger / Bypass loading** (~line 3005, right before `strata::core::MtpDrafter

    mtp;): Clear o.mtp` if an environment variable or marker file is present:
    ```cpp
    if (std::getenv("STRATA_NO_MTP") != nullptr || ::access(".no_mtp", F_OK) == 0) {
    o.mtp.clear();
    }

    (When o.mtp is empty, mtp.load() is skipped and 0 bytes are allocated for draft
    experts/head in VRAM).

    • Allow server startup without MTP (~line 4615):
    Define const bool use_mtp = !o.mtp.empty(); and relax the check:
    // change if (o.spec < 2 || o.mtp.empty() || ...) to:
    if ((use_mtp && o.spec < 2) || o.prefill_chunk <= 0 || ...)

    • Guard runtime MTP calls with use_mtp:
    • Line ~5148: (use_mtp && !mtp.bind(...)) instead of !mtp.bind(...).
    • Line ~5214 & snapshot restore/validate lines: pass use_mtp ? &mtp.kv_state() :
    nullptr instead of &mtp.kv_state().
    • Line ~5389 (sp.on_chunk): wrap sp.draft_kv / mtp.prefill inside if (use_mtp) { ...
    }.
    • Line ~6736: (use_mtp && !mtp.prefill(...)).
    • Force 1-token decode step (~line 7103 & ~line 7184):
    • In the decode loop:
    int T = use_mtp ? S_mtp : 1;
    if (first_window || !use_mtp) T = 1;

      • In the draft completion check:                                                    
    const bool drafted = !use_mtp || eos || produced_n >= max_new || mtp.draft(...);

    ──────

    2. Recompile & deploy

    ninja -C build strata
cp build/strata engine/strata

    Update engine/BUILD.json with the new source hash so setup.py doesn't overwrite it on
    start:

    python3 -c 'import setup, json, pathlib; f=pathlib.Path("engine/BUILD.json"); d=json.

    loads(f.read_text()); d["src"]=setup.source_hash(setup.ENGINE_SOURCES); f.
    write_text(json.dumps(d, indent=1))'
    ──────

    3. Run

    Create the marker file touch .no_mtp (or export STRATA_NO_MTP=1) and start ./setup.sh.

    The engine will log:
    strata serve: MTP draft layer is disabled (0 MiB VRAM used)
    and will run in pure autoregressive 1-token decode mode, leaving the extra ~850 MB VRAM
    entirely for the expert cache / context window.

  10. Niko1221 commented on Oct 13, 2026

    @Niko1221
    Niko1221
    on Oct 13, 2026
    Owner

    Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.

    If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.

  11. Niko1221
    closed this as not plannedon Oct 13, 2026

  12. lollo78 commented on Oct 19, 2026

    @lollo78
    lollo78
    on Oct 19, 2026
    Author

    Hi @Niko1221,
    I tried to run Qwen3.8 Flash-Next (Coder) also with 0.1.45 release but I have always the same issue described in the OP.

    I tried also with the "new" model supported, Qwen3.6-35B, but it fails.
    In attached the logs.

    log.txt

    Thanks for all

  13. lollo78 commented on Oct 20, 2026

    @lollo78
    lollo78
    on Oct 20, 2026
    Author

    Also 0.2.0 release fails to start on my laptop:

    Image

    Log in attached:
    strata-coder-iq1_m.log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions