Conversation
Estimate the planning kernel's shared-memory footprint and select the largest warp-aligned block size supported by the current device. Keep the existing 512-thread specialization for shapes that already fit, and add R=8 regression coverage for the H200 boundary and large expert counts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
现在一些新型 MoE(如 Mobius、MoRE、MoUE)开始通过跨层共享 Expert,把原本分散在不同深度上的专家容量合并成更宽的全局/共享 Expert Pool,因此单次路由可面对的 Expert 数会显著增加,达到数千级。
MoonEP 的 planning kernel 目前固定使用 512 个线程,也就是 16 个 warp。其中最大的 scratch 项为:
此外,kernel 还会在 shared memory 中分配三个长度为
E的完整数组:s_hist、s_bp和s_col。在固定 512 个线程时,这些分配会随着E近似线性增长,最终超过单个 block 的 shared-memory 上限。以 H200 为例,其单个 block 可使用的 opt-in shared memory 上限为 232,448 B。原版在
E=2880时需要 232,272 B,仍可运行;到E=2888时增加到 232,976 B,已经无法编译或启动。E=6144时,原版需要的 shared memory 达到 494,992 B。修改方案
本 PR 根据当前设备的 shared-memory 容量动态选择 planning block:
E、R、num_sms和block_dim预估 kernel 的 shared-memory 用量,计算过程与 kernel 内的实际分配及对齐规则保持一致。block_dim。如果只保留一个 warp(32 个线程)时仍然超过设备上限,则让生成的 kernel 直接报告资源超限。
正确性验证
测试环境:8× H200。
每个 rank 的结果均为
23 passed, 1 skipped。所有成功运行的 planning 输出在 8 个 rank 上均与独立 PyTorch reference 完全一致,所有 planning invariant 检查也全部通过。大 Expert 数边界
原本能够使用 512-thread block 的 shape 在修改后仍选择 512。对这些模型分组进行配对测试后,耗时变化中位数位于
-0.11%到+0.16%之间,原有性能基本不受影响。性能测试
SMEM为 benchmark 记录的 shared-memory 字节数。精度列写“与 PyTorch reference 完全一致”,表示 8 个 rank 的全部输出元素都与 reference 相同。没有执行 reference 检查时,不推断该版本的精度通过。“模型 / 倍率”(
Model / scale)表示测试 shape 的来源和 Expert Pool 的扩展倍率。例如,Kimi-K2.5 E×8表示以 Kimi-K2.5 的模型参数为基础,仅将 Routed Expert 数量扩大到原配置的 8 倍;TopK、H等参数保持原模型配置,token 数使用表中的S。这些倍率用于模拟跨层共享后更宽的 Expert Pool,不表示对应公开模型本身采用了该 Expert 数量。下面是附件的全量测试(含 kimi系列 DeepSeek-V4-Pro GLM-5.2 GLM-5.3-Flash Qwen3.5)
全量测试.md