You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Live cache rebuild grows MoE before shrinking KV/mamba, so a resize that passes the fit check can OOM #643
Engine.rebuild_runtime_cache (engine/engine.py:1006-1021) always resizes MoE, then KV, then mamba, and each pool frees only its own old tensors. A live resize that moves memory from KV or mamba into MoE therefore allocates the larger MoE cache while the old KV/mamba pools are still resident. The fit check (validate_rebuild, engine.py:972) prices only the target geometry, so on a nearly full card a resize that passes it can OOM and roll back. (A separate rollback failure after such an OOM is #526.)
Reproduce: serve an offloaded-MoE model near full, then ft ctl cache --moe <larger> --kv <smaller> with a target total at or below the current one. A CPU check that drives rebuild_runtime_cache with fake pools under a shared cap (current moe=40, kv=50; target moe=60, kv=30; cap 100) fails on the MoE allocation while kv=50 is still allocated.
Expected: apply the shrinking resizes before the growing ones, so any geometry that passes the fit check is reachable live.
Hi @gdevenyi Fix is up in #650: the rebuild now applies shrinking resizes before growing ones, so the transient peak stays within max(current, target) and any plan that passes the fit check completes. Reproduced your scenario on a 5090 with Qwen3.8 (offload MoE, 1.97 GiB free): on main the MoE grow OOMs with the old KV resident and the rollback latches failed; on the branch the same ft ctl cache rebuild completes and serving continues. Please let me know if this fixes your issue on your card!
Included in the 1080 Ti deployment branch, rebuilt today on main 9c635a1 (torch 2.14.1+cu126, Triton 3.8, GTX 1080 Ti sm_61, Qwen3.8-Flash-Next-NVFP4 at 262K); tests/scheduler/test_cache_rebuild.py passes on sm_61. The deployment never rebuilds its caches live, so this is unit-test coverage only.
Activity
void-mckenzie commentedon Oct 8, 2026
Hi @gdevenyi Fix is up in #650: the rebuild now applies shrinking resizes before growing ones, so the transient peak stays within max(current, target) and any plan that passes the fit check completes. Reproduced your scenario on a 5090 with Qwen3.8 (offload MoE, 1.97 GiB free): on main the MoE grow OOMs with the old KV resident and the rollback latches failed; on the branch the same
ft ctl cache rebuildcompletes and serving continues. Please let me know if this fixes your issue on your card!gdevenyi commentedon Oct 13, 2026
Included in the 1080 Ti deployment branch, rebuilt today on main 9c635a1 (torch 2.14.1+cu126, Triton 3.8, GTX 1080 Ti sm_61, Qwen3.8-Flash-Next-NVFP4 at 262K);
tests/scheduler/test_cache_rebuild.pypasses on sm_61. The deployment never rebuilds its caches live, so this is unit-test coverage only.