[Ti-Tai Wang (titaiwangms)]

Summary

Expose the existing internal GEGLU activation through the public CUDA MoE and integer QMoE operators, with explicit tanh-GELU semantics and fused gate/up layouts.

The current upstream main has an internal ActivationType::Geglu, but MoEBase rejects activation_type="geglu" and restricts nonzero swiglu_fusion to SwiGLU. This prevents an ONNX caller from using the existing gated implementation for this activation.

Changes

• Accept activation_type="geglu" with explicit fused FC1 layouts: mode 1 is interleaved [gate0, up0, ...], mode 2 is chunked [gate | up]. Reject separate FC3 inputs and unfused mode 0.
• Select CUTLASS GELU_taylor for GELU(gate, approximate="tanh") * up in both gated activation dispatch paths. Keep ordinary GELU and existing SwiGLU behavior unchanged.
• Correct the chunked GEGLU gate/up and bias offsets while preserving the legacy SwiGLU ordering.
• Treat GEGLU as gated for MoE/QMoE shape validation and integer FC1 geometry, but do not route it through fast paths that hardcode SwiGLU.
• Restrict QMoE GEGLU to integer quantization; explicitly reject FP4/NVFP4/FP8/WFP4AFP8 rather than silently selecting the wrong activation.
• Add independent numerical and invalid-layout tests, and document the CUDA-only contract and precision limitations.

The port is based on 3b4a0fff56c9b89f644acfc6613ae4dad3aeb5ff. It preserves the newer QMoE INT3 schema factory and its type/shape inference implementation; schema edits are documentation-only.

Validation

• Current-main port: scoped lintrunner/clang-format and git diff --check passed.
• Earlier patched build based on 942356fce6bdc78c016e4b902e49492fdf695453: 12 selected CUDA provider tests executed, 12 passed, 0 skipped on A100/SM80.
• The GEGLU numerical tests cover float32/float16/bfloat16 MoE and float16/bfloat16 INT4 QMoE, fusion modes 1/2, token counts 1/17, and top-k 1/2. An invalid fusion-0 case checks explicit rejection.
• The reference is computed independently with std::tanh, with inputs that distinguish it from erf-GELU. FP32 uses absolute tolerance 3e-5, relative tolerance zero, and asserts erf/tanh reference separation greater than ten times that tolerance. FP16/BF16 have explicit dtype-specific tolerances.
• CPU fallback is disabled in the CUDA operator tests. The BF16 INT4 case is guarded for the relevant non-quick-build configuration.

This is a Draft: the clean current-main port has not been rebuilt or executed locally. The earlier test result is not presented as execution of this PR head. SM90/TMA, additional GPU architectures, and a fresh full-model qualification remain unverified.

Accuracy and performance limitations

These changes enable a previously unreachable public operator path; they are not a claim that the complete Gemma4 model is numerically qualified or faster.

• Fused activation/product computation uses float before the final output conversion. Eager PyTorch can materialize GELU in FP16/BF16 before multiplication, so bitwise equivalence is not promised.
• Specialized SwiGLU-only GEMV/prefill implementations remain excluded. This PR adds no GEGLU-specific performance kernel and makes no throughput claim.
• Existing full-model HF logit/KV parity failures remain failed and unwaived. MMLU-Pro generation scores do not clear those gates.
• The observed cross-runtime accuracy/performance differences have not been attributed to this patch. Matching-input activation/kernel comparisons and CUDA profiling are needed to distinguish kernel/rounding effects from quantization, attention/export differences, and generation truncation.

This PR does not change CPU or other execution providers, rebuild llama.cpp/GenAI, or change model weights/quantization recipes.