You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
--moe-cache-auto asserts "cache budget too small" instead of falling back to prefill overlap off #642
plan_cache_budget (engine/cache_budget.py:72-74) turns on prefill overlap, and with it a 2*num_experts slot floor, whenever the slot cap allows it, without checking budget_bytes. When the budget fits num_experts slots plus the KV reserve but not 2*num_experts, the plan is clamped up to the floor, and the total <= budget_bytes assert (line 89) fires instead of the overlap-off fallback at line 81. This happens after the full expert-bank load, and the error doesn't mention --disable-moe-prefill-overlap, which would fit.
Implementation (on my fork, with a test): 27fe671. When the slot floor does not fit, the plan drops prefill overlap instead of asserting. Fork PR: gdevenyi#20
Included in the 1080 Ti deployment branch, rebuilt today on main 9c635a1 (torch 2.14.1+cu126, Triton 3.8, GTX 1080 Ti sm_61, Qwen3.8-Flash-Next-NVFP4 at 262K) as part of the upstream rebase; tests/engine/test_cache_budget.py passes on sm_61. This deployment's flags don't hit the fallback (its slot floor fits), so the fix is covered by the unit test only, not by serving.
Activity
gdevenyi commentedon Oct 13, 2026
Included in the 1080 Ti deployment branch, rebuilt today on main 9c635a1 (torch 2.14.1+cu126, Triton 3.8, GTX 1080 Ti sm_61, Qwen3.8-Flash-Next-NVFP4 at 262K) as part of the upstream rebase;
tests/engine/test_cache_budget.pypasses on sm_61. This deployment's flags don't hit the fallback (its slot floor fits), so the fix is covered by the unit test only, not by serving.