Skip to content

--moe-cache-auto asserts "cache budget too small" instead of falling back to prefill overlap off #642

Open

Description

@gdevenyi

plan_cache_budget (engine/cache_budget.py:72-74) turns on prefill overlap, and with it a 2*num_experts slot floor, whenever the slot cap allows it, without checking budget_bytes. When the budget fits num_experts slots plus the KV reserve but not 2*num_experts, the plan is clamped up to the floor, and the total <= budget_bytes assert (line 89) fires instead of the overlap-off fallback at line 81. This happens after the full expert-bank load, and the error doesn't mention --disable-moe-prefill-overlap, which would fit.

from freetoken.engine.cache_budget import plan_cache_budget
a = (2557061534, 2764800, 1683456, 512, 24576)
plan_cache_budget(*a, True, 128, 24576)   # AssertionError: cache budget too small (moe=1024 slots)
plan_cache_budget(*a, False, 128, 24576)  # (846, 129, False)

Expected: fall back to overlap off, and log it, when 2*num_experts slots plus the KV reserve don't fit.

Environment: main 07813248 (source build), RTX 5080 16 GB (sm_120), driver 610.57.04, Ubuntu 24.04, Python 3.14, torch 2.11.0+cu130, triton 3.6.0; checkpoint nvidia/Qwen3.8-Flash-Next-NVFP4.

Implementation (on my fork, with a test): 27fe671. When the slot floor does not fit, the plan drops prefill overlap instead of asserting. Fork PR: gdevenyi#20

Activity

  1. gdevenyi commented on Oct 13, 2026

    @gdevenyi
    ContributorAuthor

    Included in the 1080 Ti deployment branch, rebuilt today on main 9c635a1 (torch 2.14.1+cu126, Triton 3.8, GTX 1080 Ti sm_61, Qwen3.8-Flash-Next-NVFP4 at 262K) as part of the upstream rebase; tests/engine/test_cache_budget.py passes on sm_61. This deployment's flags don't hit the fallback (its slot floor fits), so the fix is covered by the unit test only, not by serving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions