Repository navigation
FreeToken Roadmap (2026) #79
Description
FreeToken RoadMap - 2026
New Model Support
- GLM-5.3-Flash @andy-yang-1 feat(glm5_next): support GLM-5.3-Flash #332
- Qwen3.8-Flash-Next @jason-fxz
- pin PLE in RAM — landed in feat(qwen4_exp): support Qwen3.8-Flash-Next #257
- offload PLE to Disk feat(qwen4_exp): stream the PLE n-gram table from disk #311
- DeepSeek-V4.1-Flash feat(deepseek-v41): support text and vision serving #603
- swa-encoder-bounded-replay
Hardware Support
- macOS: a native Metal engine on Apple Silicon Macs.
- AMD: support for AMD GPUs via ROCm. AMD Roadmap 2026 Q4-2027 Q1 #541
- DGX Spark: aarch64 wheels, sm_121 kernels, and a unified-memory mode.
Multimodal
- Support Image input for vision-language models
- FTW support & fix old-FTW fix(checkpoint): serve and repair FTW checkpoints around the vision encoder #486
- Qwen3.5 / Qwen3.6 / Qwen3.8 / Qwen3.8-Flash-Next family feat(mm): serve image input on the Qwen families #454
- Gemma-4 family feat(gemma4): serve image input on the Gemma-4 releases #467
- GLM5.3 Flash feat(glm5_next): serve image input on GLM-5.3-Flash #479
- DeepSeek V4 Flash vision exp feat(deepseek_v4): serve image input on the V4-Flash-Vision release #502
- Muse-Glimmer feat(muse_glimmer): serve image input on Muse-Glimmer-30B #481
- MiniMax-M3 feat(minimax_m3): serve image input on MiniMax-M3 #480
- DeepSeek-V4.1-Flash feat(deepseek-v41): support text and vision serving #603
Multi-GPU
- DGX Spark: multi-GPU optimisation for linked DGX Spark units.
- serve one model across multiple GPUs on one machine.
Quantization
- Quant Layer Refactor:
- MoE cache and CPU expert refactor to support more quantization MoE cache and CPU expert refactor #671
- GGUF: support GGUF checkpoints and their quantization types across model architectures.
Speculative Decoding
- Speculative decoding, including MTP / DFlash / Dspark.
Any contribution is welcome. Join the Developer Slack if you want to get involved.
TODO: This is still WIP.
Metadata
Metadata
Assignees
Labels
No labels
Activity
dusty44 commentedon Aug 23, 2026
Any thoughts about enabling experts to stream from NVME drives? with the cached experts optimised in three tiers vram, system ram then streamed from disk? I've built it in a dwarfstar derived engine and it can work shockingly well.
Aladincz commentedon Aug 24, 2026
Is there any chance that this technology could be usable and give some benefits to old macs running small LLM on CPU? Would it be technically applicable?
Headmaster218 commentedon Aug 24, 2026
Interesting timing — I’ve independently been working on a fairly similar MoE serving problem on AMD RDNA3/Vulkan, mainly around expert caching/paging, heterogeneous CPU–GPU execution and large-model offloading.
It’s still very much WIP and I’m cleaning up the benchmarks/docs, but the current code is here: [infr]
Running on a single RX 7900 XTX + 64GB DDR4. Happy to compare notes on the AMD side.
Successfully running with
Qwen3.6-35B-A3B with decode at 150 ~ 40tk/s, prefill at 3000 ~ 400tk/s,
Qwen3.5-122B-A10B with decode at ~23tk/s
Ling 3.0 flash with decode at 36tk/s,
Deepseek V4 Flash at ~4tk/s
orangelightening commentedon Aug 24, 2026
Looking forward to the qwen3.8 27b model support. I am using the qwen3.6 35b in the freetoken engine and using deepseek harness to serve it via the openai compatible endpoint. I have an rx3060 and 128gb of ram on an old xeon server and this is the best performing local ai system I have owned. Absolutely wonderful.
Headmaster218 commentedon Aug 25, 2026
Different from MoE, 27B is a kind of dense model, that doesnt suit this kind of cache architecture. But, Qwen team said they will public a new middle size MoE model this week, probably Qwen3.8-122B-A10B, (so thats why now im optimizing Qwen3.5-122B-A10B ;) ), and probably it will be more powerful than Qwen3.8 27B, so just, wait for it.
erikfenriz commentedon Aug 25, 2026
Would be so awesome to see it on macOS!
McL-M commentedon Aug 26, 2026
First, let me just say this is very interesting work. Keep it up.
Unfortunately, I can't try it. Lot's of older AMD GPUs can't run ROCm.
But they can use Vulkan. And so can virtually everything else.
Please consider supporting Vulkan!
Headmaster218 commentedon Aug 26, 2026
https://github.com/Headmaster218/infr
Try MoE4All ;) , it uses Vulkan
Abdul-Sen commentedon Aug 26, 2026
Tried with Qwen3.8-Flash-Next?
Headmaster218 commentedon Aug 26, 2026
yes!!! just had an early success!!!!! Qwen3.8-Flash-Next!!!
McL-M commentedon Aug 27, 2026
Yeah, I'll check it out! Thanks! 😁
radubalanpro commentedon Aug 27, 2026
Any plans for AMD gfx1151 ? I have a GMKtec EVO-X2, Ryzen AI Max+ 395 3.0GHz, 128GB DDR5, Radeon 8060S. I am running llama.cpp with ROCm on linux - custom build (actually wsl2) with pretty good results in inference, better then Vulkan in Windows, but i look forward to FreeToken, As for a test i get 24 t/s on Qwen3.8-27 Q4KM gguf and around 35-40 t/s on qwen-coder-next UD Q4KL unsloth
LeonardoDaviti commentedon Aug 27, 2026
Same, but Vulkan for support will be much easier no?
skywalk1411 commentedon Aug 28, 2026
maybe my build works on gfx1151, this works on gfx1150 #241
dbourdea commentedon Sep 2, 2026
I have created a fork/port of the original for AMD Strix Halo. I have 2 GMKtec EVO-X2s with 64GB ram (practically stole them from the person i got them from ($USD 950 each). I am running Ubuntu natively and have ditched windows completely. Both of these and an HX370 based Beelink AI 9 computer run inference at full context and 2 slots with the vulkan/RADV/llama.cpp stack. That is until i ported Freetoken to the Strix Halo last week. I switched to ROCm 10 (just came out last week) and the latest version of HIP. once I got it stable and producing at least equal to the ROCm / HIP / Freetoken stack, i created the below white paper. I've posted the paper and additional info on my github. I am continuing to research and refine the AMD port and would be open to collaborating.
Below is a link to the paper and how to access the setup.
https://github.com/dbourdea/freetoken-amd-strix-halo-paper
maxwhipw commentedon Sep 4, 2026
Suggesting to prioritize vulkan over RocM for AMD
From my research and experience right now on an AMD I am seeing more far performance on Vulkan, and vulkan has the advantage of working on more hardware.
Hardware I'm using is Strix Halo AI Max+ 395
Yuepixel commentedon Sep 18, 2026
Small note on the
Tensor parallelismitem — there is already a working port ahead of it, in case it is useful as a reference or a starting point.We got FreeToken TP=2 running for NVFP4 quantized MoE experts on a heterogeneous consumer CUDA box: RTX 3090 Ti (sm_86, 24 GB) + RTX 4060 Ti (sm_89, 16 GB) on WSL2. Neither card has native FP4, so the NVFP4 experts execute as W4A16. Model: Qwen3.6-35B-A3B-NVFP4. Verification is greedy output bit-exact vs TP=1 (5/5 prompts, temperature 0).
What the patch actually had to touch:
tp_ok=Falserejection, shardintermediateper rank, and let the existing_maybe_all_reducefold the partial experts (nvfp4.py,nvfp4_banks.py).out_proj→ row-parallel (gdn.py).o_proj→ row-parallel (attention.py).q|k|vtensors; a naive row-halving keeps the shapes valid but produces fluent-looking garbage..clone()on every slice view, otherwisenarrow()pins the whole parent allocation and you leak host memory per rank.Full write-up + the patch: #520
Reproduction repo (patch, scripts, verification harness, benchmark numbers): https://github.com/Yuepixel/freetoken-tp2-nvfp4
Credit: this is a port of the methodology from #478 (2×RX 9070 XT, ROCm, FP8) to CUDA / NVFP4 / heterogeneous cards. Related upstream work: #70.
Honest caveat: on a heterogeneous pair TP=2 decode was ~2x slower than the single 3090 Ti (the slowest GPU gates each step), but it raised the KV/context budget from 8,198 to 272,785 tokens per rank. So this is a feasibility/capacity win, not a speed win, on this particular box.