[Akshay Sonawane (apsonawane)]

Summary

Optimize four-row/top-10 packed INT2 QMoE verification without changing default dispatch.

• Add ORT_QMOE_INT2_DECODE_MAX_EXPANDED_ROWS, default 20; opt in to 40 expanded rows for four-row/top-10 GEMV.
• Add ORT_INT2_S4_FUSED_FINALIZE=1 to fuse eligible INT2 FC2 GEMV and routing finalization, removing the intermediate output traffic and separate finalize launch.
• Instantiate FP16/BF16 INT2 fused kernels and preserve activation-dtype rounding before expert weighting.
• Add a four-row/top-10 dense-reference parity test across fused/unfused modes and FP16/BF16.
• Exclude rejected wider-CTA experimental tiling and unrelated cache-placement changes.

Base and integration

Incremental PR against asonawane/main-ops-quantized-moe-fp8, not main, which lacks prerequisite operators. Companion components are the S4 grouped-attention, hyperconnection, and GenAI DFlash2 device-pipeline branches. No assets, binaries, benchmark/evaluation scripts, or Mobius changes are included.

Validation

• Changed GEMV CUDA, QMoE dispatch, and C++ reference-test translation units compile with this branch's contrib headers against the existing integration core build.
• git diff --check passes; changed lines formatted with clang-format.
• Original combined experiment: 24 numeric reference tests passed. Not a full clean build or numeric execution of these rebased operator branches. This PR remains draft pending exact-branch numeric qualification.

Combined H200 experiment (not a MoE-only ablation)

INT2 target + FP8 DFlash2, greedy batch one; six-run historical means:

Context	Decode tokens/s	TTFT ms
2K	333.964	580.497
8K	342.140	2421.365
16K	339.684	5030.850
32K	336.767	10520.556

Peak sampled memory: 59.350 GiB versus 64.322 GiB old-runtime control. These measurements belong to the previously tested combined private integration, not an isolated benchmark of this rebased PR.

Accuracy caveat: paired HumanEval DFlash/same-S4 perturbed-draft control scored 154/164 with exact output parity. Five-problem LiveCodeBench smoke scored 2/5 versus historical MTP 3/5, including a capacity-limited regression. Accuracy qualification remains open.

Companion draft PRs

• GenAI device pipeline and integration: Integrate the DFlash2 device pipeline with Qwen graph execution onnxruntime-genai#2771
• Grouped INT8 S4 attention: Enable grouped INT8 sparse attention for four-row verification #33546
• Hyperconnection S4 tuning: [CUDA] Optimize HyperConnectionLowRankMix for S<=8 speculative decode and deterministic prefill #33548

[Copilot]

🟡 Changes recommended

The parity tests can hide decode-dispatch regressions and do not detect loss of the required intermediate rounding.

2 open issues

• Test can pass without exercising the GEMV path · New
• Parity test misses activation-dtype rounding loss · New

What changed in this PR

Adds opt-in CUDA QMoE optimizations for four-row, top-10 INT2 verification while preserving default dispatch.

Changes:

• Allows an expanded-row limit of 40 and optional fusion of FC2 computation with routing finalization.
• Instantiates FP16/BF16 fused INT2 kernels with activation-dtype rounding before expert weighting.
• Adds parity tests and documents the opt-in controls.

File	Description
onnxruntime/​test/​contrib_ops/​moe_test.cc	Adds four-row/top-10 parity cases for FP16 and BF16.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_quantization.cc	Selects eligible fused FC2 finalization.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_gemv.cu	Adds configurable row admission, INT2 rounding, and fused instantiations.
docs/​contrib_ops/​cuda/​moe_qmoe.md	Documents opt-in behavior and parity coverage.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Exact-branch GPU numerical qualification remains pending for the fused FP16/BF16 execution path.

1 open issue

• Clarify error message for values below or above expanded-row limits · New

2 resolved since last review

• Parity test misses activation-dtype rounding loss
• Test can pass without exercising the GEMV path

_{Review effort: Balanced}